Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Duet of Representations and How Explanations Exacerbate It

Charles Wan, Rodrigo Belo, Leid Zejnilović, Susana Lavado

TL;DR

This paper investigates how explanations of AI predictions interact with humans' preexisting beliefs, showing that explanations can worsen decision quality when they foreground conflicting priors. It formalizes human priors and algorithmic causal representations and treats explanations as compressed representations that shape attention, testing these ideas in a field experiment with SHAP explanations in a public employment service context using an XGBoost predictor. Using LASSO-based methods to identify conflicts (e.g., college education conflicting with LTU risk) and regression analyses to assess effects on decision quality and confidence, it finds that exposing conflicting features degrades performance, especially when counselors adjust predictions. The work argues for a broader notion of communicative rationality in human-AI interaction, offering desiderata—understanding, reciprocity, negotiability, and shared reality—to bridge epistemic gaps and improve practical outcomes.

Abstract

An algorithm effects a causal representation of relations between features and labels in the human's perception. Such a representation might conflict with the human's prior belief. Explanations can direct the human's attention to the conflicting feature and away from other relevant features. This leads to causal overattribution and may adversely affect the human's information processing. In a field experiment we implemented an XGBoost-trained model as a decision-making aid for counselors at a public employment service to predict candidates' risk of long-term unemployment. The treatment group of counselors was also provided with SHAP. The results show that the quality of the human's decision-making is worse when a feature on which the human holds a conflicting prior belief is displayed as part of the explanation.

The Duet of Representations and How Explanations Exacerbate It

TL;DR

This paper investigates how explanations of AI predictions interact with humans' preexisting beliefs, showing that explanations can worsen decision quality when they foreground conflicting priors. It formalizes human priors and algorithmic causal representations and treats explanations as compressed representations that shape attention, testing these ideas in a field experiment with SHAP explanations in a public employment service context using an XGBoost predictor. Using LASSO-based methods to identify conflicts (e.g., college education conflicting with LTU risk) and regression analyses to assess effects on decision quality and confidence, it finds that exposing conflicting features degrades performance, especially when counselors adjust predictions. The work argues for a broader notion of communicative rationality in human-AI interaction, offering desiderata—understanding, reciprocity, negotiability, and shared reality—to bridge epistemic gaps and improve practical outcomes.

Abstract

An algorithm effects a causal representation of relations between features and labels in the human's perception. Such a representation might conflict with the human's prior belief. Explanations can direct the human's attention to the conflicting feature and away from other relevant features. This leads to causal overattribution and may adversely affect the human's information processing. In a field experiment we implemented an XGBoost-trained model as a decision-making aid for counselors at a public employment service to predict candidates' risk of long-term unemployment. The treatment group of counselors was also provided with SHAP. The results show that the quality of the human's decision-making is worse when a feature on which the human holds a conflicting prior belief is displayed as part of the explanation.
Paper Structure (17 sections, 5 equations, 2 figures, 4 tables)

This paper contains 17 sections, 5 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Theory and Related Work
  3. Human-algorithm Collaboration
  4. Affective States
  5. The Duet of Representations
  6. Explanations as Compressed Representations
  7. Psychological Salience and Information Processing
  8. Field Experiment
  9. Setting
  10. Research Ethics and Social Impact
  11. Methods
  12. Identifying the Conflict
  13. Identifying the Effects of Explanations on Decision Quality and Confidence
  14. Results
  15. Discussion
  16. ...and 2 more sections

Figures (2)

  • Figure 1: User Interface for the Control Group. In the "Modelo Atual" panel the risk score and the risk assessment are shown. In the "Justificação da Segemntação Atribuída" panel the counselor has to select her own risk assessment on the top and her confidence level on the right.
  • Figure 2: User Interface for the Treatment Group. SHAP are shown in the "Principais Factores" panel with their respective effect on the risk of LTU.