Impact of AI-Triage on Radiologist Report Turnaround Time: Real-World Time-Savings and Insights from Model Predictions

TL;DR

This paper addresses the variable effectiveness of AI-enabled triage for prioritizing pulmonary embolism in chest CT pulmonary angiography by accounting for real-world radiologist workflow. It combines a large retrospective dataset (n=11,252) with a queueing-based simulation (QuCAD) to predict time-savings in radiologist report turnaround time under different workflow conditions, such as disease prevalence, inter-arrival times, and staff availability. The key findings show significant work-hour TAT reductions ($22.2$ minutes, $p=0.004$) but not during off-hours, with QuCAD predicting $29.6$ minutes and $2.10$ minutes for work and off-hours, respectively, aligning with observed data. The study highlights that small changes in workflow can invert conclusions about AI benefits and provides a framework to forecast AI-triage impact in diverse clinical settings.$68.9$,$46.7$,$44.8$,$42.0$,$22.2$,$29.6$,$2.10$

Abstract

Objective: To quantify the impact of workflow parameters on time-savings in report turnaround time (TAT) due to an AI-triage device that prioritized pulmonary embolism (PE) in chest CT pulmonary angiography (CTPA) exams. Methods: This retrospective study analyzed 11252 adult CTPA exams conducted for suspected PE at a single tertiary academic medical center. Data was divided into two periods: pre-AI and post-AI. For PE-positive exams, TAT - defined as the duration from patient scan completion to the first preliminary report completion - was compared between the two periods. Time-savings were reported separately for work-hour and off-hour cohorts. To characterize radiologist workflow, 527234 records were retrieved from the PACS and workflow parameters such as exam inter-arrival time and radiologist read-time extracted. These parameters were input into a computational model to predict time-savings following deployment of an AI-triage device and to study the impact of workflow parameters. Results: The pre-AI dataset included 4694 chest CTPA exams with 13.3% being PE-positive. The post-AI dataset comprised 6558 exams with 16.2% being PE-positive. The mean TAT for pre-AI and post-AI during work hours are 68.9 [95% CI" 55.0, 82.8] and 46.7 [38.1, 55.2] minutes respectively, and those during off-hours are 44.8 [33.7, 55.9] and 42.0 [33.6, 50.3] minutes. Clinically-observed time-savings during work hours (22.2 [95% CI: 5.85, 38.6] minutes) were significant (p=0.004), while off-hour (2.82 [-11.1, 16.7] minutes) were not (p=0.345). Observed time-savings aligned with model predictions (29.6 [95% range: 23.2, 38.1] minutes for work hours; 2.10 [1.76, 2.58] minutes for off-hours). Discussion: Consideration and quantification of clinical workflow contribute to an accurate assessment of the expected time-savings in TAT following deployment of an AI-triage device.

Figures (4)

• Figure 1: Flowchart depicting the number of adult chest CTPA exams before and after the AI-triage device was implemented. AI = Artificial Intelligence; CTPA = CT Pulmonary Angiography.
• Figure 2: Results of workflow parameters from data for modeling.
• Figure 3: An ROC curve using the sensitivity and specificity reported in the FDA 510(k) summary under a bi-normal assumption. ROC = Receiver Operating Characteristic; AUC = Area Under the ROC Curve; FDA = U.S. Food and Drug Administration.
• Figure 4: Results of clinically-observed and model-predicted time-savings.