Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Over-Relying on Reliance: Towards Realistic Evaluations of AI-Based Clinical Decision Support

Venkatesh Sivaraman, Katelyn Morrison, Will Epperson, Adam Perer

TL;DR

This paper critiques the dominant use of trust, reliance, acceptance, and task-performance (TRAP) metrics in AI-based clinical decision support (AI-CDS) evaluations, arguing they overlook other forms of value and how clinicians actually use AI. It advocates ecologically valid, domain-appropriate study designs that assess real-world workflows, downstream tasks, and emergent uses of AI beyond the AI's explicit decision task. By detailing four TRAP-related pitfalls and calling for broader, interdisciplinary evaluation approaches, the authors aim to align AI-CDS research with practical clinical benefits and complementarity between clinicians and AI. The analysis emphasizes AI's value in information retrieval, workflow coordination, communication, and support for care teams, not solely improvements in predictive accuracy.

Abstract

As AI-based clinical decision support (AI-CDS) is introduced in more and more aspects of healthcare services, HCI research plays an increasingly important role in designing for complementarity between AI and clinicians. However, current evaluations of AI-CDS often fail to capture when AI is and is not useful to clinicians. This position paper reflects on our work and influential AI-CDS literature to advocate for moving beyond evaluation metrics like Trust, Reliance, Acceptance, and Performance on the AI's task (what we term the "trap" of human-AI collaboration). Although these metrics can be meaningful in some simple scenarios, we argue that optimizing for them ignores important ways that AI falls short of clinical benefit, as well as ways that clinicians successfully use AI. As the fields of HCI and AI in healthcare develop new ways to design and evaluate CDS tools, we call on the community to prioritize ecologically valid, domain-appropriate study setups that measure the emergent forms of value that AI can bring to healthcare professionals.

Over-Relying on Reliance: Towards Realistic Evaluations of AI-Based Clinical Decision Support

TL;DR

This paper critiques the dominant use of trust, reliance, acceptance, and task-performance (TRAP) metrics in AI-based clinical decision support (AI-CDS) evaluations, arguing they overlook other forms of value and how clinicians actually use AI. It advocates ecologically valid, domain-appropriate study designs that assess real-world workflows, downstream tasks, and emergent uses of AI beyond the AI's explicit decision task. By detailing four TRAP-related pitfalls and calling for broader, interdisciplinary evaluation approaches, the authors aim to align AI-CDS research with practical clinical benefits and complementarity between clinicians and AI. The analysis emphasizes AI's value in information retrieval, workflow coordination, communication, and support for care teams, not solely improvements in predictive accuracy.

Abstract

As AI-based clinical decision support (AI-CDS) is introduced in more and more aspects of healthcare services, HCI research plays an increasingly important role in designing for complementarity between AI and clinicians. However, current evaluations of AI-CDS often fail to capture when AI is and is not useful to clinicians. This position paper reflects on our work and influential AI-CDS literature to advocate for moving beyond evaluation metrics like Trust, Reliance, Acceptance, and Performance on the AI's task (what we term the "trap" of human-AI collaboration). Although these metrics can be meaningful in some simple scenarios, we argue that optimizing for them ignores important ways that AI falls short of clinical benefit, as well as ways that clinicians successfully use AI. As the fields of HCI and AI in healthcare develop new ways to design and evaluate CDS tools, we call on the community to prioritize ecologically valid, domain-appropriate study setups that measure the emergent forms of value that AI can bring to healthcare professionals.

Paper Structure

This paper contains 3 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Locating the Traps in Trust-and-Reliance Evaluations
  3. The Way Forward: Identifying and Measuring AI's Value to Clinicians

Figures (1)

  • Figure 1: Many prior studies measure AI's success in clinical contexts in terms of their trust, reliance, acceptance, or performance on a task aligned with the AI's training. However, this fails to capture other forms of value that AI can provide to clinicians.