DC Public Safety Radio
See the main site at http://media.csosa.gov
See the radio program at http://media.csosa.gov/podcast/audio/2016/01/predicting-criminal-behavior-through-risk-instruments/
Leonard Sipes: From the nation’s capital, this is DC Public Safety. I’m your host Leonard Sipes. Today’s program, ladies and gentlemen, Predicting Criminal Risk. We’re going to be taking a look at risk instruments. What are they? How good are they? From the Washington State Institute for Public Policy we have Zachary Hamilton, Assistant Professor Department of Criminal Justice and Criminology. Director of Washington State Institute for Criminal Justice at the Washington State University.
Joining Zachary will be Mason Burley, Senior Research Associate Washington State Institute for Public Policy. The website www.wsipp.wa.gov. Gentlemen welcome to DC Public Safety.
Male: Hello. Thank you for having us.
Leonard Sipes: All right, risk assessments. This is something that seems to be is the foundation of meaningful change within the criminal justice system. We now within the court services and and offender supervision agency, my agency, we’ve been using risk instruments for about 10 years. Risk instruments are being used for sentencing, they’re being used in all things inside the criminal justice system. Give me, first of all, whether or not you think risk instruments are the foundation of meaningful change within criminal justice. Then give me a layman’s definition of what risk instruments are.
Z. Hamilton: Well I can jump in on that. I do think risk assessments are the foundation of really how we move forward in the criminal justice system. A lot of what people have complained about in years past has sort of been the inconsistency, inaccuracy, and maybe the overuse of discretion within the system. What risk assessments do is they put everyone on an even playing field, or at least they attempt to put everyone on an even playing field, so that everyone’s judged in a similar manner. In terms of what a risk assessment is, in terms of layman’s language, it’s essentially a set of items that you use. Maybe a survey or a questionnaire, or maybe they’re items that are collected from a mix of self reported questionnaires or criminal history measures that are known within an agencies records.
These items usually contain a mix of what are called static items, that look at the offenders criminal history, their age, their gender maybe. But also some dynamic items too that try to focus on the offenders needs. Trying to examine whether or not they have issues with regards to employment, substance abuse, medical health, residency, any sort of things that may directly or indirectly impact their future criminal behavior or misbehavior on supervision.
Leonard Sipes: So we’re looking at criminal history, we’re looking at the age of the offender. We’re talking about possibly the sex of the offender. There are static issues as well as dynamic issues that go into this that formulate a sense as to who this person is, and what their level of risk is, and what their level of need is in terms of social services?
Z. Hamilton: That is correct and some of what you’re describing in terms of how risk assessments and needs assessments have been extended throughout the criminal justice system is about the idea that there’s certain behaviors that we would like to predict. Recidivism is one but there’s others that you had mentioned that may exist at a different point within the system. The individual coming into the system may have a first risk or needs assessment completed at the pre-trial phase. Maybe a judge might use that to identify whether or not this person would be a risk of flight. At that point you might be trying to predict recidivism but you might also be trying to predict failure to appear.
Once the persons incarcerated you might want to predict infraction behavior. Once they’ve been released or re-entered into the community you’d want to predict recidivism behavior but maybe you’d also like to predict compliance while on supervision. There’s a multitude of outcomes in which you can examine and utilize these tools to create essentially prediction models.
Leonard Sipes: What we have is law enforcement using them, pre-trial using them, the judge using them, in terms of sentencing. Parole and probation agencies using them to figure out how closely to watch this individual. Correctional systems, prisons, could be using them in terms of the potential for good behavior within the prison system. Across the board risk instruments are becoming a bigger part of the criminal justice system.
Z. Hamilton: Yes as more individuals are gathering data and more agencies are gathering data. Tracking offenders and identifying how you can utilize that data to better supervise, more efficiently supervise, or even remove individuals from supervision to create a better system is, I think, where risk assessment is heading.
Leonard Sipes: I do want to go back to Dr. Hamilton in terms of discussion as to how effective these instruments are, but Mason Burley I’m going to go to you. This whole issue of risk assessment instruments started with the insurance industry, did it not? The insurance industry for decades, multiple decades, has been assessing the risk of individuals on their caseloads, if you will, as to whether or not they’re going to be a risk in terms of health. Whether or not they’re going to have a heart attack. Whether or not they’re going to be unemployed. Whether or not they’re going to have health problems across the board. Am I right or wrong?
M. Burley: I think actuarial instruments have been used in a lot of different arenas. Certainly outside the public policy area and I wanted to emphasize… Insurance would certainly be one of those, but the history in Washington State really goes back to quite an extensive and evolving use of risk assessment in the criminal justice area, if I can focus on that for a second. Because I think the history’s pretty important in terms of the utility we found in the state. Initially in the late 90’s the juvenile justice system decided to look at some evidence based approaches and to target who might be best served in some of those programs they developed risk assessment instruments for all the juvenile courts in the state.
That later evolved into the supervision that Dr. Hamilton mentioned with the department of corrections and then looking at who was at most risk of further crime and looking at supervision resources and how those should be targeted to high risk individuals. That risk assessment has gone through several iterations and Dr. Hamilton and his institute really refined and improved as they’ve learned from what works with best with an instrument. It’s been adopted in other areas as he mentioned throughout the state with pre-trial and judges deciding information about bail and pre-trial decisions based on historic risk.
Then finally the work we recently released that the institute looked at, the mental health population and what individuals who have been involved in the forensic mental health system and the type of risk they pose. I think that it’s becoming more recognized, the value of it in Washington State and we provide a good test case in some of these scenarios as individuals move through different phases of the criminal justice system.
Leonard Sipes: Before continuing I do want to plug the Washington State Institute for Public Policy as putting out some of the most easy to read, clear, precise, research findings. Not just within the criminal justice system but across all phases of the government. I do want to congratulate the Washington State Institute for Public Policy for it’s dedication to put out the research that the rest of us, who are not researchers can understand and the policy makers can grasp and run with. You guys have probably over a decade of experience putting out nice, clear, and concise public policy research. Again their website www.wsipp.wa for the state of washington .gov.
Did you want to continue Dr. Hamilton?
Z. Hamilton: I think Mason covered a lot of what’s been done but it’s going back several years. Even back to 1997, when the Washington State Institute of Public Policy mainly one of their key researchers, Robert Bernowsky, created one of the first juvenile risk assessments that was used throughout the state. That spawned this idea of collecting data overtime, tracking offender populations and re-adjusting those assessment models to improve prediction over time. As people change, as the population changes, as even the criminal statues change. The focus of the assessment is fine tuned over time and that’s something that was put into place early with WSIPP. I think those traditions are starting to continue on now with these new adjustments to the adult risk tools as well.
Leonard Sipes: There’s an endless list of policy questions I do want to get into in terms of Microsoft coming out with an app that’s predicting future criminal behavior. Commercial applications that law enforcement is now using. The attorney general of the United States, Attorney General Holder, former Attorney General who criticized risk instruments used in sentencing about possible bias. Every time that there is a mass shooting there is a psychologist that gets on CNN and says that there is no way we can predict future criminal behavior. All of those are issues that I want to get on to or discuss. The biggest issue that people when they come to me and talk to me is, “Leonard, how effective are these things.” That’s why I love your research.
You mentioned the fact that you did something recent talking about the criminal population within the state of Washington but also whether or not the involuntary treatment population for mental health reasons. Whether or not the risk instruments that the state of Washington was using could be used for both groups. So I’m using that as the basis for this program. While you say while no risk instrument can predict future criminal offenses with 100% accuracy. The goal is to create an assessment that has strong predictive performance. How strong is that predictive performance?
Z. Hamilton: There’s different industry standards for how we identify predictive performance and as you mentioned you probably don’t want me to go too far down the rabbit hole in terms of giving a statistics lesson. The common metric of which people base an assessment is what’s called an area under the curve statistic. There’s really ways of identifying the strength of the instrument and industry standards that set cut points within the statistic to say what’s a weak prediction, what’s a moderate prediction, what’s a strong prediction. What we’re finding with these tools is that we’ve advanced our methods as we gather more data. As we’re able to refine and focus on specific types of crimes. Not just any recidivism generally but maybe focusing on what predictors predict violent crime versus property crime or drug crime. You’re able to really hone in on that prediction and get strong models almost every single time.
Without going too much into the detail of what those industry… We tend to exceed the industry standard for what are called strong models and a lot of our models tend to give in to those upper echelons of being able to accurately identify recidivism prediction across the population at rates of over 70%.
Leonard Sipes: Well over 70% would be astounding and I think that gives individuals a fairly decent benchmark in terms of understanding risk instruments. In other words there’s no way that we can do this with 100% predictive behavior. That’s impossible, but at the 100% level that’s pretty [dag-gone 00:12:20] good and pretty predictive. I want to get into the categories used in the research. What they tried to do was to focus on four particular categories. This was the division of correction there in the state of Washington. High Violent, High Non-Violent, Moderate Risk or Low Risk. They tried to keep it simple in terms of falling into one of those four categories, correct?
Z. Hamilton: Yes. Really it’s a big advance and they’ve been doing it for a while in Washington State but it’s a distinction that Washington State has that I believe is a big advantage as compared to other risk assessment instruments. Anybody who supervises the centers will tell you it’s not just the probability of any risk, it’s the type of risk that they pose. Many risk assessment instruments will essentially say, “Are you low, moderate, or high risk? What’s your probability of risk for committing a new arrest or a new conviction.”
That’s great but somebody that has, let’s just say, a 45% likelihood of committing a drug crime, versus an individual that has a 42% likelihood of committing a violent crime. Yeah the percentages are different but you’re going to supervise those individuals differently. The severity or the public perception of a particular crime is going to be of note. So an individual that may have a slightly lower probability of committing a violent crime may be supervised at a greater rate simply because the threat to society or to public safety is a little bit stronger than that person that’s more likely to commit a drug crime.
Is that coming through okay?
Leonard Sipes: Yeah, perfectly. That’s why I wanted to start off with the issue of fundamental change within the criminal justice system. Because it seems, in terms of evidence based practices, what we’re seeing is that we should be focusing our resources on the highest risk offenders and not focusing our resources on lower risk offenders. Because we’re talking about 5 million human beings on any given day under community supervision currently. Under community supervision on any given day according to US Department of Justice Data and Parole and Probation agencies throughout the country. They can have rations of 100:1, 200:1, I’ve seen 250:1. Luckily here in Washington DC our maximum caseload is 50:1, for specialized cases it’s much lower than that. But when you have that disparity between say, in terms of community supervision prone probation agents and enormous case loads, you’ve got to figure out who’s your highest risk and provide the resources to that highest risk offender. Correct?
Z. Hamilton: That is correct and one of the distinctions within Washington State, and this has been going on ever since 2007 I believe. It might even be 2005. They had a statute that went through the legislature called the Offender Accountability Act. What it essentially said was we’re going to use a risk assessment to determine whose lowest risk and those lowest risk offenders are essentially not going to be supervised. If we can determine what their probability of recidivism and it’s within a range of being of low or very low risk then we don’t feel it’s within our due diligence to give them extensive supervision. There’s a fair amount of research out there that identifies the individuals that are of low risk of recidivism, the more you supervise them actually the more likely you are to observe behavior and they end up becoming more likely to commit crimes simply because of these observation effects.
Leonard Sipes: We end up re-incarcerating the wrong people. That’s the bottom line.
Z. Hamilton: That is the bottom line. What Washington State has done and has been doing for years is essentially saying administrative supervision or no supervision for those individuals that are of these lower tiered risks. That not only has sort of, fell in line with risk need and responsivity theory but it’s also saved the state a lot of money. Evaluations of this change in statute has essentially identified no uptake in recidivism following it’s passing. The effect has essentially been a net win for the state.
Leonard Sipes: We’re more than half way through the program. I do want to re-introduce our guest Zachary Hamilton, Assistant Professor Department of Criminal Justice and Criminology. Director of the Washington State Institute for Criminal Justice at the Washington State University.
Mason Burley is also by our microphone. Senior Research Associate Washington State Institute of Public Policy. Once again I will continue to praise the Washington State Institute of Public Policy for putting out extraordinarily good research. Some of the best in the United States. Www.wsipp.wa.gov.
Let me go into a little bit more about this individual research and then talk more about policy questions. What you did with this research was take a look at violent felony convictions, non-violent felony convictions, and any conviction over the course of a 2 year period to measure the accuracy of the risk instrument used for he state of Washington. You took all of that and basically you said that in some cases the degree of probability, one as high as 70%, and that was for the non-violent felony convictions. Was that correct?
M. Burley: Washington State has a long standing history of using risk assessment with the department of corrections and the prison population that are under supervision. So we’re able to kind of look at the risk elements that we use for that population and see if the same risk assessment is a valid tool for other populations as well. As you mentioned we looked at violent felony and non-violent felony for DOC and for the DOC population between the highest risk offenders 60 to 70 percent of those have a non-violent, repeat crime of a non-violent felony within 2 years.
Leonard Sipes: But those are the people that you designated in that category and the results we’re validated by saying that 70% of the people that we put into that category did have a non-violent conviction.
M. Burley: Yes, we looked at that category of the prison population under supervision and compared individuals in the mental health system in Washington State to see if the same kind of elements can be used to predict two year recidivism as well. Now for that population the recidivism rates we’re much lower. Two to three times lower in some circumstances. The risk assessment tool was still valid in that we could distinguish between low, moderate and high risk offenders along that continuum of risk.
Leonard Sipes: But what I’m asking is, is that the paper basically says that there was a … I’m simplifying things. An above 70% accuracy rate. What does that mean when you say it’s an above .75 accuracy rate? Which is to me, as a lame person, that basically says 75% of the time we were accurately able to predict. Am I right or wrong?
M. Burley: I misunderstood the question. Dr. Hamilton maybe you want to jump in.
Z. Hamilton: Yeah, it’s a little more nuanced than that. The way that you perceived that, what you’re determining the accuracy rate as is the area under the curve statistic. Essentially what it says, and I’m going to explain it as hopefully as simply as possible. If you have two groups and you separated your two groups of people you we’re observing into those that recidivated and those that did not. If you were to randomly select one person out of each one of those groups. Using this risk assessment you’d identify that the individual that recidivated had a higher risk over 70% of the time.
Leonard Sipes: Had a higher risk over 70% of the time.
Z. Hamilton: Correct.
Leonard Sipes: That doesn’t mean that they went out and under your criteria, or the criteria of the study, and were convicted within a two year time period, that they had that higher probability.
Z. Hamilton: No, no. What we do is we take that risk score. We create a continuous risk score from zero to wherever it ends up being at it’s highest. With that continuous risk score then you essentially dissect it into several pieces where you have a low, a moderate, a high non-violent and a high violent. In any one of those groups you can identify what’s the probability of somebody who falls into those categories recidivating.
Leonard Sipes: For the layman’s question ‘How accurate are these instruments?’ and using the example of what happened in the state of Washington. Is there a layman-esq answer to say that they would be accurate 70% of the time, 50% of the time, 60% of the time?
Z. Hamilton: Again it’s a little more nuanced than that. The instrument doesn’t come out and say, “This person is going to recidivate.” It doesn’t come out with a yes or no answer and say, “This person is going to recidivate. This other person, person B, is not going to recidivate.” What it does is the score will provide a probability of recidivism. Let’s say the score ranges from 0 to 100. Somebody that scores out at a 50 may have a 30% likely hood of recidivism. If that 30% likely hood puts them in the upper tier or high risk category than that category can then be identified as having their aggregate probability of recidivism.
It gets a little more complicated but essentially what we do is we utilize that area under the curve statistic to essentially rate that continuous risk score, to say how accurate it is and it also allows us to compare our instrument to someone else’s instrument. But to give a quick and easy answer, to say this persons going to recidivate and this person’s not. How accurate is the assessment? Risk assessments aren’t built to do that. They’re built to provide guidelines for individuals to say who is higher risk as compared to another person who might be of moderate or lower risk.
Leonard Sipes: But it’s inevitable that there are going to be false positives and false negatives. It’s going to be inevitable that there are a certain number of people who are designated as high risk are not going to come back to the criminal justice system. There is a certain inevitable … It is inevitable that a person that you would designate as low risk would come back to the criminal justice system. There has got to be a certain understanding by the public that these are not perfect predictive analysis. That there are going to be false positives and false negatives.
Z. Hamilton: That is true. I believe either in the report, or in one of the appendices of the report, we identified the probability of recidivism by falling into one of the many categories we’ve created the cut points for. So you can identify what’s the probability for recidivism for high violent, high non-violent, moderate and low.
M. Burley: I think I was answering that question rather than the overall predictive ability of the model. It think it’s important that what I learned from this, working with Dr. Hamilton as well is that the risk is on a continual scale and we we’re able to kind of look at … Even though there are false positives and false negatives. The likely hood of being able to tell which offenders are going to recidivate or which individuals are going to recidivate increases on a gradual basis as you move from low to high risk based on what you find in the assessment.
Leonard Sipes: I’m going to go back to questions I posed right before the break. We have everybody and their uncle now putting out risk instruments of one shape or another. Microsoft came out with an app. There are commercial entities that are basically saying that law enforcement agencies that we’re going to be able to tell you with higher degrees of probability who on the street is going to commit further crimes or commit violent crimes. We have a world that is now moving towards predictive risk instruments beyond criminal justice. The private sector is doing this. Do you have any concerns about this because it is inevitable that again we have false positives, we have false negatives. We’re going to be pin pointing people and talking about their probabilities for coming into the criminal justice system and we’re going to be wrong.
M. Burley: Yeah. I have lots of concerns about that. Not necessarily that it’s a private sector doing that. There’s plenty of companies that exist in the private sector that create great risk assessment instruments. My fear is individuals from the private sector potentially taking large data sources not knowing how exactly they fit within the jurisdiction that they’re evaluating and essentially spitting out a model that is accurate to a degree, but that accuracy isn’t really developed within the known quantities of that particular jurisdiction. Every single location in the United States is slightly different. You do have a certain stability in terms of certain items being predictive. Age being one of them, prior convictions being another. But individuals that are creating risk assessment instruments that don’t have knowledge of that on the ground usage, or the variations in the population, could potentially create models that are not as accurate as their claim.
Leonard Sipes: Every time there is a mass shooting. Every time there is a horrific violent crime in this country a psychologist will do an interview for CNN and say, “Even though the person had a history of mental health treatment. Even though the person had a history of schizophrenia …” I do want to point out that even though there are a higher percentage of the people that come from mental health backgrounds involved in the criminal justice system the overwhelming majority of people who have mental health backgrounds are not going to be coming into the criminal justice system but a psychologist or psychiatrist will stand up on CNN and say, “It is impossible to predict future criminal behavior. Yes he had contact with the mental health system, but to predict this level of violence is just literally impossible.”
Then media will pick up the phone and call me and say, “If these individuals can not be … If you can not predict their future criminality then is it … Psychologists are saying it’s impossible to predict future violent criminal behavior. Then the risk instruments that you talk about, what good are they?”
Do you see the level of confusion that folks in the media and the general public would have when a psychologist gets up and makes a statement like that?
Z. Hamilton: Yes. I can. The issue is that the risk assessment instruments that were discussing and that we’ve created, they’re built for a specific population. They’re built for people that have contact with the criminal justice system. If nobody’s had any contact with the criminal justice system they’ve never been assessed for risk. That’s one limitation right there. The other is that there’s individuals that typically commit these crimes a lot of times you’ll see those psychiatrists come up and say, “They have a mental illness, or an undiagnosed mental illness.” Again if there’s no data to be able to identify any of this persons prior behavior which is a lot of what risk assessments are built upon then it’s difficult to assess somebody’s risk.
Again going into the general population and identify someones risk of recidivism is usually not what risk assessments are built for. They’re built for release decisions, pre-trial decisions. Decisions on probation or parole and supervision. They’re not necessarily built for that particular purpose. To be even more blunt they’re built on an aggregate population so we’re addressing the aggregate risk or the average risk of a person within the population that we’ve had assessments for.
That individual that commits the serious offense or a mass shooting, that had never entered into one of those populations to be assessed. You’re not going to have any identification of risk for that particular individual and those events are so rare that they cant be predicted based on the average events that normally criminals and offenders commit.
Leonard Sipes: Our guests today have been Dr. Zachary Hamilton, Assistant Professor Department of Criminal Justice and Criminology. Director of the Washington State Institute for Criminal Justice at the Washington State University. Mason Burley’s been by our microphone. Senior Research Associate Washington State Institute for Public Policy. www.wsipp.wa.gov.
Ladies and Gentlemen this is D.C. Public Safety, we appreciate your comments. We even appreciate your criticisms and we want everybody to have themselves a very pleasant day.
