Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Generative AI Support Patients' & Caregivers' Informational Needs? Towards Task-Centric Evaluation Of AI Systems

Shreya Rajagopal, Jae Ho Sohn, Hari Subramonyam, Shiwali Mohan

TL;DR

This work argues that evaluating generative AI for healthcare should focus on patient- and caregiver-facing information needs rather than generic benchmark accuracy. It combines a formative need-finding study with radiologists to identify informational themes and develops an evaluation paradigm that pairs visual radiology data with natural-language queries to test AI systems. By comparing two state-of-the-art multimodal models (Claude-3.5-Sonnet and ChatGPT-4o) against radiologist responses across both quantitative similarity and qualitative Gricean criteria, the authors reveal substantial variability and notable gaps in correctness, relevance, and usefulness. The findings underscore the importance of task-centric, human-centered evaluation to responsibly deploy AI tools that augment, rather than replace, clinicians, and chart concrete directions for future, larger-scale studies and better evaluation frameworks.

Abstract

Generative AI systems such as ChatGPT and Claude are built upon language models that are typically evaluated for accuracy on curated benchmark datasets. Such evaluation paradigms measure predictive and reasoning capabilities of language models but do not assess if they can provide information that is useful to people. In this paper, we take some initial steps in developing an evaluation paradigm that centers human understanding and decision-making. We study the utility of generative AI systems in supporting people in a concrete task - making sense of clinical reports and imagery in order to make a clinical decision. We conducted a formative need-finding study in which participants discussed chest computed tomography (CT) scans and associated radiology reports of a fictitious close relative with a cardiothoracic radiologist. Using thematic analysis of the conversation between participants and medical experts, we identified commonly occurring themes across interactions, including clarifying medical terminology, locating the problems mentioned in the report in the scanned image, understanding disease prognosis, discussing the next diagnostic steps, and comparing treatment options. Based on these themes, we evaluated two state-of-the-art generative AI systems against the radiologist's responses. Our results reveal variability in the quality of responses generated by the models across various themes. We highlight the importance of patient-facing generative AI systems to accommodate a diverse range of conversational themes, catering to the real-world informational needs of patients.

Can Generative AI Support Patients' & Caregivers' Informational Needs? Towards Task-Centric Evaluation Of AI Systems

TL;DR

This work argues that evaluating generative AI for healthcare should focus on patient- and caregiver-facing information needs rather than generic benchmark accuracy. It combines a formative need-finding study with radiologists to identify informational themes and develops an evaluation paradigm that pairs visual radiology data with natural-language queries to test AI systems. By comparing two state-of-the-art multimodal models (Claude-3.5-Sonnet and ChatGPT-4o) against radiologist responses across both quantitative similarity and qualitative Gricean criteria, the authors reveal substantial variability and notable gaps in correctness, relevance, and usefulness. The findings underscore the importance of task-centric, human-centered evaluation to responsibly deploy AI tools that augment, rather than replace, clinicians, and chart concrete directions for future, larger-scale studies and better evaluation frameworks.

Abstract

Generative AI systems such as ChatGPT and Claude are built upon language models that are typically evaluated for accuracy on curated benchmark datasets. Such evaluation paradigms measure predictive and reasoning capabilities of language models but do not assess if they can provide information that is useful to people. In this paper, we take some initial steps in developing an evaluation paradigm that centers human understanding and decision-making. We study the utility of generative AI systems in supporting people in a concrete task - making sense of clinical reports and imagery in order to make a clinical decision. We conducted a formative need-finding study in which participants discussed chest computed tomography (CT) scans and associated radiology reports of a fictitious close relative with a cardiothoracic radiologist. Using thematic analysis of the conversation between participants and medical experts, we identified commonly occurring themes across interactions, including clarifying medical terminology, locating the problems mentioned in the report in the scanned image, understanding disease prognosis, discussing the next diagnostic steps, and comparing treatment options. Based on these themes, we evaluated two state-of-the-art generative AI systems against the radiologist's responses. Our results reveal variability in the quality of responses generated by the models across various themes. We highlight the importance of patient-facing generative AI systems to accommodate a diverse range of conversational themes, catering to the real-world informational needs of patients.
Paper Structure (31 sections, 8 figures, 1 table)

This paper contains 31 sections, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Preliminaries
  4. Formative Study of Patient-Provider Interaction
  5. Study
  6. Materials
  7. Protocol
  8. Conducting the study
  9. Inductive Thematic Analysis
  10. Theme 1: Statistics of Lung Injury and Associated Damage to Other Organs
  11. Theme 2: Role of Members of the Medical Team
  12. Theme 3: Locating Issues mentioned in the Radiology Report to the CT Scan
  13. Theme 4: Selecting a Treatment Plan
  14. Theme 5: Understanding Medical Terminology from Radiology Report
  15. Theme 6: Diagnostic Follow-up Plan
  16. ...and 16 more sections

Figures (8)

  • Figure 1: Patient journey through the medical system highlight various interactions they have with different healthcare experts.
  • Figure 2: 2 axial slices of the lung CT and the radiology report presented to study participants
  • Figure 3: Protocol showing the context and task prompt as well as participant actions and interactions
  • Figure 4: Stacked bar chart describing the proportion of each theme in each participant’s interaction with the radiologist, with the portion of the conversation occupied by each theme on the y-axis, and participants along the x-axis.
  • Figure 5: A heatmap plotting the semantic similarity of responses generated by Claude and ChatGPT under two experimental configurations to an expert's response.
  • ...and 3 more figures