Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Evaluating Explanation Utility for Human-AI Decision Making in NLP

Fateme Hashemi Chaleshtori, Atreya Ghosal, Alexander Gill, Purbid Bambroo, Ana Marasović

TL;DR

The results show that providing AI predictions, with or without explanations, does not cause decision makers to speed up their work without compromising performance, and it is argued for revisiting the setup of human-AI teams and improving automatic deferral of instances to AI, where explanations could play a useful role.

Abstract

Is explainability a false promise? This debate has emerged from the insufficient evidence that explanations help people in situations they are introduced for. More human-centered, application-grounded evaluations of explanations are needed to settle this. Yet, with no established guidelines for such studies in NLP, researchers accustomed to standardized proxy evaluations must discover appropriate measurements, tasks, datasets, and sensible models for human-AI teams in their studies. To aid with this, we first review existing metrics suitable for application-grounded evaluation. We then establish criteria to select appropriate datasets, and using them, we find that only 4 out of over 50 datasets available for explainability research in NLP meet them. We then demonstrate the importance of reassessing the state of the art to form and study human-AI teams: teaming people with models for certain tasks might only now start to make sense, and for others, it remains unsound. Finally, we present the exemplar studies of human-AI decision-making for one of the identified tasks -- verifying the correctness of a legal claim given a contract. Our results show that providing AI predictions, with or without explanations, does not cause decision makers to speed up their work without compromising performance. We argue for revisiting the setup of human-AI teams and improving automatic deferral of instances to AI, where explanations could play a useful role.

On Evaluating Explanation Utility for Human-AI Decision Making in NLP

TL;DR

The results show that providing AI predictions, with or without explanations, does not cause decision makers to speed up their work without compromising performance, and it is argued for revisiting the setup of human-AI teams and improving automatic deferral of instances to AI, where explanations could play a useful role.

Abstract

Is explainability a false promise? This debate has emerged from the insufficient evidence that explanations help people in situations they are introduced for. More human-centered, application-grounded evaluations of explanations are needed to settle this. Yet, with no established guidelines for such studies in NLP, researchers accustomed to standardized proxy evaluations must discover appropriate measurements, tasks, datasets, and sensible models for human-AI teams in their studies. To aid with this, we first review existing metrics suitable for application-grounded evaluation. We then establish criteria to select appropriate datasets, and using them, we find that only 4 out of over 50 datasets available for explainability research in NLP meet them. We then demonstrate the importance of reassessing the state of the art to form and study human-AI teams: teaming people with models for certain tasks might only now start to make sense, and for others, it remains unsound. Finally, we present the exemplar studies of human-AI decision-making for one of the identified tasks -- verifying the correctness of a legal claim given a contract. Our results show that providing AI predictions, with or without explanations, does not cause decision makers to speed up their work without compromising performance. We argue for revisiting the setup of human-AI teams and improving automatic deferral of instances to AI, where explanations could play a useful role.
Paper Structure (48 sections, 15 figures, 20 tables)

This paper contains 48 sections, 15 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Review of Application-Grounded Explanation Evaluation
  3. Taxonomy of explanation evaluation.
  4. Reliance definitions.
  5. Measuring reliance.
  6. Deferral.
  7. Analysis of Task Appropriateness
  8. Task Criteria
  9. Categorization of ExNLP Tasks
  10. Are ExNLP tasks connected to real-world applications beyond debugging?
  11. Do ExNLP tasks require notable human effort? Are people skilled at solving these tasks?
  12. Are ExNLP tasks associated with high-risk situations?
  13. Task Checks with Model Performance
  14. Study I: People Make All Decisions
  15. Study Design
  16. ...and 33 more sections

Figures (15)

  • Figure 1: An overview of the five task presentations, P1--P5, used for Study I (see §\ref{['sec:task_conditions']}).
  • Figure 2: Instructions for Study I (§\ref{['sec:study1_main']}), the experiments with conditions P1 to P3.
  • Figure 3: Instructions for Study I (§\ref{['sec:study1_main']}), the experiment with conditions P4.
  • Figure 4: Instructions for Study I (§\ref{['sec:study1_main']}), the experiment with conditions P5.
  • Figure 5: Instructions for expert performance estimation experiment.
  • ...and 10 more figures