Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-AI Collaboration in Radiology: The Case of Pulmonary Embolism

Paul Goldsmith-Pinkham, Chenhao Tan, Alexander K. Zentefis

TL;DR

The paper investigates real-world human–AI collaboration in radiology for suspected pulmonary embolism by analyzing 117,063 CTPA scans read by 389 radiologists across eight sites during a staggered rollout of an FDA-cleared AI tool. It demonstrates asymmetric agreement with AI (97% for AI-negative vs 84% for AI-positive predictions), evolving disagreement over time, and substantial heterogeneity in radiologist engagement and override behavior, including a nonmonotonic relationship where moderate AI engagement yields the highest concordance. Despite a doubling of radiologist reading volumes, diagnostic speed remains stable and mortality outcomes do not worsen, suggesting AI primarily enhances workflow efficiency through better triage and management of negative cases. The findings illuminate automation bias versus learning in high-stakes medical decisions and highlight the importance of accounting for cross-physician heterogeneity, gender differences, and engagement patterns when deploying AI in clinical practice. These insights have practical implications for optimizing AI-assisted radiology deployment, training, and governance to maximize safety, efficiency, and clinician trust in high-stakes decision support systems.

Abstract

We study how radiologists use AI to diagnose pulmonary embolism (PE), tracking over 100,000 scans interpreted by nearly 400 radiologists during the staggered rollout of a real-world FDA-approved diagnostic platform in a hospital system. When AI flags PE, radiologists agree 84% of the time; when AI predicts no PE, they agree 97%. Disagreement evolves substantially: radiologists initially reject AI-positive PEs in 30% of cases, dropping to 12% by year two. Despite a 16% increase in scan volume, diagnostic speed remains stable while per-radiologist monthly volumes nearly double, with no change in patient mortality -- suggesting AI improves workflow without compromising outcomes. We document significant heterogeneity in AI collaboration: some radiologists reject AI-flagged PEs half the time while others accept nearly always; female radiologists are 6 percentage points less likely to override AI than male radiologists. Moderate AI engagement is associated with the highest agreement, whereas both low and high engagement show more disagreement. Follow-up imaging reveals that when radiologists override AI to diagnose PE, 54% of subsequent scans show both agreeing on no PE within 30 days.

Human-AI Collaboration in Radiology: The Case of Pulmonary Embolism

TL;DR

The paper investigates real-world human–AI collaboration in radiology for suspected pulmonary embolism by analyzing 117,063 CTPA scans read by 389 radiologists across eight sites during a staggered rollout of an FDA-cleared AI tool. It demonstrates asymmetric agreement with AI (97% for AI-negative vs 84% for AI-positive predictions), evolving disagreement over time, and substantial heterogeneity in radiologist engagement and override behavior, including a nonmonotonic relationship where moderate AI engagement yields the highest concordance. Despite a doubling of radiologist reading volumes, diagnostic speed remains stable and mortality outcomes do not worsen, suggesting AI primarily enhances workflow efficiency through better triage and management of negative cases. The findings illuminate automation bias versus learning in high-stakes medical decisions and highlight the importance of accounting for cross-physician heterogeneity, gender differences, and engagement patterns when deploying AI in clinical practice. These insights have practical implications for optimizing AI-assisted radiology deployment, training, and governance to maximize safety, efficiency, and clinician trust in high-stakes decision support systems.

Abstract

We study how radiologists use AI to diagnose pulmonary embolism (PE), tracking over 100,000 scans interpreted by nearly 400 radiologists during the staggered rollout of a real-world FDA-approved diagnostic platform in a hospital system. When AI flags PE, radiologists agree 84% of the time; when AI predicts no PE, they agree 97%. Disagreement evolves substantially: radiologists initially reject AI-positive PEs in 30% of cases, dropping to 12% by year two. Despite a 16% increase in scan volume, diagnostic speed remains stable while per-radiologist monthly volumes nearly double, with no change in patient mortality -- suggesting AI improves workflow without compromising outcomes. We document significant heterogeneity in AI collaboration: some radiologists reject AI-flagged PEs half the time while others accept nearly always; female radiologists are 6 percentage points less likely to override AI than male radiologists. Moderate AI engagement is associated with the highest agreement, whereas both low and high engagement show more disagreement. Follow-up imaging reveals that when radiologists override AI to diagnose PE, 54% of subsequent scans show both agreeing on no PE within 30 days.
Paper Structure (24 sections, 2 equations, 10 figures, 9 tables)

This paper contains 24 sections, 2 equations, 10 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Setting and Data
  3. Pulmonary Embolism
  4. AI-Assisted Diagnosis
  5. Data Sources
  6. Electronic Health Records
  7. AI Platform Developer
  8. National Provider Registries
  9. Linking Radiologists Across Data Sources
  10. Descriptive Statistics
  11. Care Sites and Diagnosis Rates
  12. Patient Characteristics
  13. Scan Characteristics
  14. Radiologist Characteristics
  15. Scan Efficiency
  16. ...and 9 more sections

Figures (10)

  • Figure 1: Pulmonary Embolism Anatomy
  • Figure 2: Computed Tomographic Pulmonary Angiography (CTPA) of Pulmonary Embolism
  • Figure 3: Monthly CTPA Scan Volume
  • Figure 4: Monthly PE Positivity Rate
  • Figure 5: Radiologist Agreement Rates with AI by AI Diagnosis
  • ...and 5 more figures