Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Patient Safety Risks from AI Scribes: Signals from End-User Feedback

Jessica Dai, Anwen Huang, Catherine Nasrallah, Rhiannon Croci, Hossein Soleimani, Sarah J. Pollet, Julia Adler-Milstein, Sara G. Murray, Jinoos Yazdany, Irene Y. Chen

TL;DR

The study investigates patient safety risks associated with AI scribes by analyzing end-user feedback from a large U.S. hospital system. It employs a mixed-methods design combining quantitative per-encounter feedback analysis (BERTopic/Sentence-BERT) and qualitative provider surveys to identify safety signals and understand clinician perceptions. Findings indicate safety concerns related to medication transcription, HPI/A/P documentation, and other note-formation processes, though heterogeneity among clinicians and the limited sample size limit prevalence estimates. The authors argue that end-user feedback can serve as a real-time monitoring signal for deployment of AI scribes and highlight methodological considerations for handling one-sided feedback. The work lays groundwork for automated safety-signal systems in clinical AI deployments.

Abstract

AI scribes are transforming clinical documentation at scale. However, their real-world performance remains understudied, especially regarding their impacts on patient safety. To this end, we initiate a mixed-methods study of patient safety issues raised in feedback submitted by AI scribe users (healthcare providers) in a large U.S. hospital system. Both quantitative and qualitative analysis suggest that AI scribes may induce various patient safety risks due to errors in transcription, most significantly regarding medication and treatment; however, further study is needed to contextualize the absolute degree of risk.

Patient Safety Risks from AI Scribes: Signals from End-User Feedback

TL;DR

The study investigates patient safety risks associated with AI scribes by analyzing end-user feedback from a large U.S. hospital system. It employs a mixed-methods design combining quantitative per-encounter feedback analysis (BERTopic/Sentence-BERT) and qualitative provider surveys to identify safety signals and understand clinician perceptions. Findings indicate safety concerns related to medication transcription, HPI/A/P documentation, and other note-formation processes, though heterogeneity among clinicians and the limited sample size limit prevalence estimates. The authors argue that end-user feedback can serve as a real-time monitoring signal for deployment of AI scribes and highlight methodological considerations for handling one-sided feedback. The work lays groundwork for automated safety-signal systems in clinical AI deployments.

Abstract

AI scribes are transforming clinical documentation at scale. However, their real-world performance remains understudied, especially regarding their impacts on patient safety. To this end, we initiate a mixed-methods study of patient safety issues raised in feedback submitted by AI scribe users (healthcare providers) in a large U.S. hospital system. Both quantitative and qualitative analysis suggest that AI scribes may induce various patient safety risks due to errors in transcription, most significantly regarding medication and treatment; however, further study is needed to contextualize the absolute degree of risk.

Paper Structure

This paper contains 12 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. System setup and data sources
  3. Quantitative analysis
  4. Qualitative analysis
  5. Discussion
  6. Supplemental materials for quantitative analysis
  7. Details on filtering step
  8. Details on handling physician heterogeneity
  9. Additional figures illustrating heterogeneity.
  10. Condensing feedback volume for outlier physicians.
  11. Details on LLM per-cluster summary
  12. Details on LLM-overall summary

Figures (3)

  • Figure 1: Heterogeneity in feedback volume (bar chart y-axis) and rates (coloring) per physician; each bar corresponds to one unique clinician. y-axis truncated for clarity (top user submitted 151).
  • Figure 2: Feedback rates (i.e. fractions of encounters in which feedback was submitted), per user. Alternative visualization of coloring data from Figure \ref{['fig:hetero']}.
  • Figure 3: Encounters at which feedback is provided, per user. Blue dots were marked as non-safety related; red were safety-relevant. Gray dots are encounters where no text feedback was provided.