Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court

Angela Jin, Niloufar Salehi

TL;DR

This paper study efforts to contest AI systems in practice by studying how public defenders scrutinize AI in court, and provides recommendations that center the technical, social, and institutional context to better position interventions to support contestability in practice.

Abstract

Accountable use of AI systems in high-stakes settings relies on making systems contestable. In this paper we study efforts to contest AI systems in practice by studying how public defenders scrutinize AI in court. We present findings from interviews with 17 people in the U.S. public defense community to understand their perceptions of and experiences scrutinizing computational forensic software (CFS) -- automated decision systems that the government uses to convict and incarcerate, such as facial recognition, gunshot detection, and probabilistic genotyping tools. We find that our participants faced challenges assessing and contesting CFS reliability due to difficulties (a) navigating how CFS is developed and used, (b) overcoming judges and jurors' non-critical perceptions of CFS, and (c) gathering CFS expertise. To conclude, we provide recommendations that center the technical, social, and institutional context to better position interventions such as performance evaluations to support contestability in practice.

(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court

TL;DR

This paper study efforts to contest AI systems in practice by studying how public defenders scrutinize AI in court, and provides recommendations that center the technical, social, and institutional context to better position interventions to support contestability in practice.

Abstract

Accountable use of AI systems in high-stakes settings relies on making systems contestable. In this paper we study efforts to contest AI systems in practice by studying how public defenders scrutinize AI in court. We present findings from interviews with 17 people in the U.S. public defense community to understand their perceptions of and experiences scrutinizing computational forensic software (CFS) -- automated decision systems that the government uses to convict and incarcerate, such as facial recognition, gunshot detection, and probabilistic genotyping tools. We find that our participants faced challenges assessing and contesting CFS reliability due to difficulties (a) navigating how CFS is developed and used, (b) overcoming judges and jurors' non-critical perceptions of CFS, and (c) gathering CFS expertise. To conclude, we provide recommendations that center the technical, social, and institutional context to better position interventions such as performance evaluations to support contestability in practice.
Paper Structure (47 sections, 4 figures, 1 table)

This paper contains 47 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Public defense in the U.S.
  4. Computational forensic software in the U.S. criminal legal system
  5. Facial recognition
  6. Probabilistic genotyping software (PGS)
  7. Related Work
  8. Contesting automated decision systems
  9. Designing and communicating performance evaluations
  10. Designing performance evaluations
  11. Communicating performance evaluations
  12. Methodology
  13. Data collection
  14. Participant recruitment
  15. Interview protocol
  16. ...and 32 more sections

Figures (4)

  • Figure 1: We used this storyboard to establish a hypothetical scenario in which the prosecution uses evidence output by probabilistic genotyping software (PGS). We presented this scenario before presenting potential evaluation approaches depicted in Figures \ref{['fig:adv-scrut-storyboard']}, \ref{['fig:checklist-storyboard']}, and \ref{['fig:model-cards-storyboard']}. This scenario is modeled off of a real case publicly documented by kirchner2017thousands, and uses an image from the article created by Michael Hirshon for ProPublica.
  • Figure 2: This storyboard depicts a potential evaluation approach based on abebe2022adversarial.
  • Figure 3: This storyboard depicts a potential evaluation approach based on the Justice in Forensic Algorithms Act of 2021, a bill introduced to Congress by Rep. Mark Takano [D-CA-41] takano2021.
  • Figure 4: This storyboard depicts a potential evaluation approach based on mitchell2019model.