A Quantitative Framework to Predict Wait-Time Impacts Due to AI-Triage Devices in a Multi-AI, Multi-Disease Workflow

TL;DR

multi-QuCAD addresses how deploying multiple AI-triage devices affects radiologist wait-times across disease conditions. It combines a queueing-theory–based analytical framework with a simulation tool to quantify wait-time savings for target conditions and delays for non-targeted ones, under preemptive-resume and non-preemptive scheduling with priority and hierarchical protocols. The key contributions are the extended theoretical models, the simulation platform, and four brain-imaging experiments including a 4-AI, 9-disease scenario, revealing trade-offs and providing a practical tool for planning AI deployments. The work highlights the need to balance time-savings for AI-targeted diseases against potential delays for other urgent conditions, with implications for safety, equity, and operational efficiency in radiology departments.

Abstract

The deployment of multiple AI-triage devices in radiology departments has grown rapidly, yet the cumulative impact on patient wait-times across different disease conditions remains poorly understood. This research develops a comprehensive mathematical and simulation framework to quantify wait-time trade-offs when multiple AI-triage devices operate simultaneously in a clinical workflow. We created multi-QuCAD, a software tool that models complex multi-AI, multi-disease scenarios using queueing theory principles, incorporating realistic clinical parameters including disease prevalence rates, radiologist reading times, and AI performance characteristics from FDA-cleared devices. The framework was verified through four experimental scenarios ranging from simple two-disease workflows to complex nine-disease systems, comparing preemptive versus non-preemptive scheduling disciplines and priority versus hierarchical triage protocols. Analysis of brain imaging workflows demonstrated that while AI-triage devices significantly reduce wait-times for target conditions, they can substantially delay diagnosis of non-targeted, yet urgent conditions. The study revealed that hierarchical protocol generally provides more wait-time savings for the highest-priority conditions compared to the priority protocol, though at the expense of more delays to lower-priority patients with other time-sensitive conditions. The quantitative framework presented provides essential insights for orchestrating multi-AI deployments to maximize overall patient time-saving benefits while minimizing unintended delay for other important patient populations.

Figures (13)

• Figure 1: An example of how a multi-AI, multi-disease workflow is defined. Bigger boxes represent different image groups by imaging modalities, whereas smaller blue boxes are a set of disease conditions detected using the corresponding imaging modalities. A set of AIs (AI-A to AI-D) are included to identify and triage a subset of disease conditions. NCCT = Non-Contrast Computed Tomography; CTA = Computed Tomography Angiography; MRI = Magnetic Resonance Imaging; SDH = Subdural Hemorrhage; SAH = Subarachnoid Hemorrhage; LVO = Large Vessel Occlusion; DAI = Diffuse Axonal Injury; AVM = Arteriovenous malformations; AI = Artificial Intelligence.
• Figure 2: An example of the types of multi-AI, multi-disease workflow. a) A without-AI workflow has a first-in first-out (FIFO) queue in which in-coming images are reviewed in the order of their arrivals. b) In a with-AI, priority workflow, all AI-positive cases are triaged as one high-priority class (red), and AI-negative cases are in the low-priority class (blud). Within the prioritized class, images are reviewed in the order of their arrivals regardless of the time-sensitiveness among disease conditions. c) A with-AI, hierarchical workflow further prioritizes the AI-positive cases by their time-sensitiveness based on a disease hierarchy (i.e. LVO being more time-sensitive than SAH) defined by the user. Hence, the 2-AI hierarchnical workflow has three priority classes: cases flagged by AI-LVO have the highest priority (red), cases flagged by AI-SAH have a middle priority (brown), and cases that are deemed negative by both AIs have the lowest priority (blue). SAH = Subarachnoid Hemorrhage; LVO = Large Vessel Occlusion; AI = Artificial Intelligence.
• Figure 3: The AI and disease condition settings for the four experimental workflow scenarios. Each box with different shade colors represents an image group. Each group consists of patients with a disease condition as well as non-diseased (ND) patients. The smaller blue boxes represent the disease condition and ND subgroups. Each disease condition is defined by a disease name, its time-sensitiveness ranking (with 1 being the most time-critical condition), and the name of the corresponding AI-triage device (if any). LVO = Large Vessel Occlusion; SAH = Subarachnoid Hemorrhage; SDH = Subdural Hemorrhage; AI = Artificial Intelligence.
• Figure 4: Sensitivity and specificity performance of FDA-cleared AI-triage devices targeting LVO (left) and ICH (right) respectively. Each gray dot is the sensitivity and specificity pair from an AI-triage device with their 95% confidence intervals. The green star is the average performance among all devices. An ROC curve in green is obtained by fitting the average performance with a binomial assumption. ICH = Intracerebral Hemorrhage; LVO = Large Vessel Occlusion; FDA = U.S. Food and Drug Administration; ROC = Receiver Operating Characteristic; AI = Artificial Intelligence.
• Figure 5: The numerical values of the parameters for the 4-AI, 9-disease workflow scenario (Experiment 4). (Left) Groups are defined by a block with its name, group probability, disease conditions involved, and the mean read-time of the non-diseased cases in the group. Each disease is defined by four parameters: disease name, its hierarchical rank, prevalence within the group, and the mean read-time. (Right) Each AI is defined by a block with its name, the group that the AI belongs to, the target disease condition, the true-positive fraction (TPF), and the false-positive fraction (FPF). Sensitivity is the same as TPF, whereas specificity is 1-FPF. A Receiver operating characteristic (ROC) curve can also be provided via a csv file.