A Comparative User Study of Human Predictions in Algorithm-Supported Recidivism Risk Assessment

TL;DR

This study investigates how machine-learning guidance in a risk assessment instrument affects human predictions of violent recidivism. Employing a RiskEval-based predictor and a mix of crowdsourced and targeted participants, the authors assess predictive accuracy, alignment with algorithmic recommendations, and willingness to rely on automated support, complemented by focus groups with professionals. Key findings show that algorithmic help generally improves accuracy, with larger gains among domain experts and data scientists, and that numerical presentation and risk scales influence performance and trust. The work highlights the importance of human oversight, routine updates, and context-aware design for deploying RAIs in high-stakes criminal justice decisions.

Abstract

In this paper, we study the effects of using an algorithm-based risk assessment instrument to support the prediction of risk of criminalrecidivism. The instrument we use in our experiments is a machine learning version ofRiskEval(name changed for double-blindreview), which is the main risk assessment instrument used by the Justice Department ofCountry(omitted for double-blind review).The task is to predict whether a person who has been released from prison will commit a new crime, leading to re-incarceration,within the next two years. We measure, among other variables, the accuracy of human predictions with and without algorithmicsupport. This user study is done with (1)generalparticipants from diverse backgrounds recruited through a crowdsourcing platform,(2)targetedparticipants who are students and practitioners of data science, criminology, or social work and professionals who workwithRiskEval. Among other findings, we observe that algorithmic support systematically leads to more accurate predictions fromall participants, but that statistically significant gains are only seen in the performance of targeted participants with respect to thatof crowdsourced participants. We also run focus groups with participants of the targeted study to interpret the quantitative results,including people who useRiskEvalin a professional capacity. Among other comments, professional participants indicate that theywould not foresee using a fully-automated system in criminal risk assessment, but do consider it valuable for training, standardization,and to fine-tune or double-check their predictions on particularly difficult cases.

Figures (11)

• Figure 1: Sequence of studies and number of participants.
• Figure 2: Risk scales used in our experiments (left: absolute scale, right: relative scale).
• Figure 3: Average AUC with 95% confidence interval by group. See Table \ref{['tab:results1']} in Supplementary Material \ref{['ann:accuracy-per-subgroup']} for details.
• Figure 4: Average difference between human and algorithm prediction by case order, absolute scale.
• Figure 5: AUC of participant predictions before and after algorithmic support for participants who received algorithmic support (excludes control group).