"Even explanations will not help in trusting [this] fundamentally biased system": A Predictive Policing Case-Study

TL;DR

This work investigates whether different explanation modalities (text, visual, and hybrid) can cultivate appropriate trust in AI-driven predictive policing systems, comparing expert (retired police) and lay users. Across two preregistered studies with a total of 192 participants, explanations did not meaningfully improve appropriate trust, though hybrid explanations increased subjective trust for experts in the first study. Perceived usefulness of explanations predicted subjective trust but did not translate into improved decision quality, highlighting a gap between trust and performance in high-risk AI contexts. The findings imply that explanations should not be used solely to boost trust and motivate policy recommendations focused on trust calibration, human oversight, and evaluation metrics that prioritize decision quality over subjective trust in public-sector AI deployments.

Abstract

In today's society, where Artificial Intelligence (AI) has gained a vital role, concerns regarding user's trust have garnered significant attention. The use of AI systems in high-risk domains have often led users to either under-trust it, potentially causing inadequate reliance or over-trust it, resulting in over-compliance. Therefore, users must maintain an appropriate level of trust. Past research has indicated that explanations provided by AI systems can enhance user understanding of when to trust or not trust the system. However, the utility of presentation of different explanations forms still remains to be explored especially in high-risk domains. Therefore, this study explores the impact of different explanation types (text, visual, and hybrid) and user expertise (retired police officers and lay users) on establishing appropriate trust in AI-based predictive policing. While we observed that the hybrid form of explanations increased the subjective trust in AI for expert users, it did not led to better decision-making. Furthermore, no form of explanations helped build appropriate trust. The findings of our study emphasize the importance of re-evaluating the use of explanations to build [appropriate] trust in AI based systems especially when the system's use is questionable. Finally, we synthesize potential challenges and policy recommendations based on our results to design for appropriate trust in high-risk based AI-based systems.

Figures (3)

• Figure 1: Visual explanations for a selected instance from the user study.
• Figure 2: An illustration of the four steps performed by participants of the user-study. In step 1, participants rate their confidence in accurately identifying the hotspot. In step 2, the AI assistant selects a hotspot with its reasoning in form of explanations. In step 3, the participants makes their final decision (Q3). Finally, in step 4, participants rate their subjective trust, usefulness of the explanations and open questions related to decision-making.
• Figure 3: An illustration of mean responses for changes in Global Trust Meter over 8 rounds for study 1.