One VLM, Two Roles: Stage-Wise Routing and Specialty-Level Deployment for Clinical Workflows

TL;DR

The paper tackles operational bottlenecks in clinical AI by proposing a single vision–language model that serves dual roles: (i) a calibrated, stage-wise router that maps input images to the appropriate model-card (and can abstain early for efficiency), and (ii) a specialty-specific downstream model fine-tuned on multiple tasks to unify deployment. The three-stage routing pipeline (modality, primary abnormality, model-card) uses stage-wise prompts and a calibrated top-2 selector to improve routing accuracy and calibration, achieving notable gains over a baseline router ($+\approx$ $+8$ to $+11$ pp) and lower $ECE$. Separately, the authors demonstrate that the same VLM, when fine-tuned with parameter-efficient methods like QLoRA across five specialties, approaches or matches specialty baselines across a broad set of datasets, enabling a single deployment stack per specialty. Collectively, these solutions reduce data-science workload, simplify monitoring, and increase transparency via per-stage justifications and calibrated thresholds, offering a practical path from triage through deployment. The work highlights practical deployment benefits while acknowledging limitations, such as distributional shifts outside the training modalities and the need for ongoing calibration and interpretability enhancements.

Abstract

Clinical ML workflows are often fragmented and inefficient: triage, task selection, and model deployment are handled by a patchwork of task-specific networks. These pipelines are rarely aligned with data-science practice, reducing efficiency and increasing operational cost. They also lack data-driven model identification (from imaging/tabular inputs) and standardized delivery of model outputs. We present a framework that employs a single vision-language model (VLM) in two complementary, modular roles. First (Solution 1): the VLM acts as an aware model-card matcher that routes an incoming image to the appropriate specialist model via a three-stage workflow (modality -> primary abnormality -> model-card ID). Reliability is improved by (i) stage-wise prompts enabling early termination via "None"/"Other" and (ii) a calibrated top-2 answer selector with a stage-wise cutoff. This raises routing accuracy by +9 and +11 percentage points on the training and held-out splits, respectively, compared with a baseline router, and improves held-out calibration (lower Expected Calibration Error, ECE). Second (Solution 2): we fine-tune the same VLM on specialty-specific datasets so that one model per specialty covers multiple downstream tasks, simplifying deployment while maintaining performance. Across gastroenterology, hematology, ophthalmology, pathology, and radiology, this single-model deployment matches or approaches specialized baselines. Together, these solutions reduce data-science effort through more accurate selection, simplify monitoring and maintenance by consolidating task-specific models, and increase transparency via per-stage justifications and calibrated thresholds. Each solution stands alone, and in combination they offer a practical, modular path from triage to deployment.

Figures (7)

• Figure 1: The first stage in the router workflow (modality detection).
• Figure 2: The second stage in the router workflow (disease identification).
• Figure 3: The third (final) stage in the router workflow (model-card matching).
• Figure 4: Answer selector (per stage)
• Figure 5: Cutoff sweep (training split, stage $k$)