A multitask framework for automated interpretation of multi-frame right upper quadrant ultrasound in clinical decision support

TL;DR

This work addresses the operator-dependence of RUQ ultrasound interpretation by introducing a multitask vision–language agent that jointly performs abnormality classification, automated report generation, and treatment recommendations on multi-frame RUQ cine loops. The approach leverages a domain-specific Ultrasound Foundation Model backbone in a three-stage pipeline, trained on a large multi-center dataset and validated across external cohorts, with blinded expert and LLM-based evaluations. Findings show strong diagnostic realism, high information density in reports, and effective cholecystectomy prediction using AI-generated narratives, though cross-site generalization remains a challenge requiring domain adaptation. Overall, the study demonstrates the potential of end-to-end VLMs to standardize RUQ interpretation, enhance reporting efficiency, and support real-time surgical triage in emergency care, while highlighting the value of clinician–AI collaboration and the need for prospective validation.

Abstract

Ultrasound is a cornerstone of emergency and hepatobiliary imaging, yet its interpretation remains highly operator-dependent and time-sensitive. Here, we present a multitask vision-language agent (VLM) developed to assist with comprehensive right upper quadrant (RUQ) ultrasound interpretation across the full diagnostic workflow. The system was trained on a large, multi-center dataset comprising a primary cohort from Johns Hopkins Medical Institutions (9,189 cases, 594,099 images) and externally validated on cohorts from Stanford University (108 cases, 3,240 images) and a major Chinese medical center (257 cases, 3,178 images). Built on the Qwen2.5-VL-7B architecture, the agent integrates frame-level visual understanding with report-grounded language reasoning to perform three tasks: (i) classification of 18 hepatobiliary and gallbladder conditions, (ii) generation of clinically coherent diagnostic reports, and (iii) surgical decision support based on ultrasound findings and clinical data. The model achieved high diagnostic accuracy across all tasks, generated reports that were indistinguishable from expert-written versions in blinded evaluations, and demonstrated superior factual accuracy and information density on content-based metrics. The agent further identified patients requiring cholecystectomy with high precision, supporting real-time decision-making. These results highlight the potential of generalist vision-language models to improve diagnostic consistency, reporting efficiency, and surgical triage in real-world ultrasound practice.

Figures (6)

• Figure 1: Overview of the right upper-quadrant (RUQ) ultrasound workflow and the proposed US-Agent model.a, Clinical workflow of RUQ ultrasound examination, illustrating the sequential steps from clinical evaluation and ultrasound acquisition to radiologist interpretation, diagnostic reporting, and surgical decision-making. b, Workflow of the US-Agent model. The internal Johns Hopkins dataset is used for both training and inference, while external datasets from Stanford and a major Chinese medical center are used for independent testing. The trained agent supports three downstream tasks: abnormality classification, automated report generation, treatment decision guidance. A radiologist Turing test was performed to assess realism and clinical accuracy. c, Detailed structure of each downstream task. The abnormality classification module (USFM) predicts 18 diagnostic labels; the report generation module (Qwen VLM) produces structured radiology reports; the treatment decision module integrates image, report, disease, and clinical features to identify patients who require cholecystectomy; and the Turing test compares model-generated versus radiologist-written reports to assess realism and diagnostic fidelity.
• Figure 1: Analysis of Turing test performance and evaluator agreement.a | Experimental design. AI-generated radiology reports were evaluated under two settings: (1) AI & Clinician preference, in which radiologists selected between the original and AI-generated reports, and (2) AI & Clinician collaboration, in which a radiologist first edited the AI-generated reports before comparison. Three aspects were analyzed: evaluation time variance, inter-rater agreement, and the effect of rater expertise (expert vs trainee). b | Evaluation time per case. Each point represents one report evaluation ($n=60$). The AI & Clinician collaboration setting required longer average review time (mean difference $\Delta$ = 0.85 min), reflecting increased depth of review and proofreading effort. c | Inter-rater agreement. Heatmaps display the distribution of reports by the number of raters (0–6) who preferred the AI-generated report, illustrating consistency across evaluators. d | Effect of rater expertise. Agreement patterns stratified by experience show that experts achieved higher consensus and stronger alignment when reviewing AI-assisted (edited) reports compared with unedited versions. Collectively, these findings indicate that clinician–AI collaboration enhances interpretive consistency and diagnostic trust, albeit with a modest increase in evaluation time.
• Figure 2: Performance of the automated abnormality classifier and cross-dataset generalization.a, Per-label comparison of classification metrics between the Ultrasound Foundation Model (USFM) and the ResNet-50 baseline across representative organ systems. The USFM consistently outperformed the ImageNet-pretrained ResNet across all categories. b, Comparison of disease prevalence (bars, color-coded by dataset) and corresponding AUROC (lines of matching color) across the cohorts. Despite substantial variations in disease prevalence, model performance remained stable across datasets. c, Correlation between disease prevalence and AUROC for the internal and external validation cohorts. No statistically significant association was observed ($p > 0.05$), confirming the model’s robustness to label imbalance and institutional variability.
• Figure 3: Cross-institutional comparison of automated report generation performance. Bar plots summarizing the performance of UltrasoundLLama, UltrasoundPhi, and UltrasoundQwen across the internal JHU dataset and two external validation cohorts from Stanford University and a major Chinese medical center. Each metric represents the mean of three independent runs, with standard deviations shown as error bars. Text similarity metrics (BLEU, ROUGE, METEOR, BERT-F1) capture linguistic and semantic fidelity, whereas clinical realism metrics (DocLensxie2023doclens, and the FORTEli2025towards categories) assess factual accuracy and structural correspondence to expert-written reports. UltrasoundQwen consistently achieved the highest scores across all datasets. Statistical significance is denoted by asterisks (* $p<0.05$, ** $p<0.01$, *** $p<0.001$).
• Figure 4: Turing test evaluation Results. Boxplots show accuracy (a), recall (b), and F1 score (c) for radiologist judgments in both settings (blue = AI versus Clinician preference, pink = AI + Clinician collaboration) under conditions with and without ultrasound images. Collaboration and image access significantly improved performance across all metrics (*** $p < 0.001$).