Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Organ-Aware Attention Improves CT Triage and Classification

Lavsen Dahal, Yubraj Bhandari, Geoffrey D. Rubin, Joseph Y. Lo

TL;DR

ORACLE-CT tackles the challenge of calibrated CT triage by introducing organ-aware pooling that restricts evidence to anatomically meaningful regions, paired with optional Organ-Scalar Fusion to inject concise organ-level cues. The method is encoder-agnostic and evaluated under a fixed, auditable protocol on chest (CT-RATE, RAD-ChestCT) and abdomen (MERLIN) datasets, achieving state-of-the-art supervised performance and improved calibration compared with prior VLM-based and baseline approaches. Across chest and abdomen, organ-masked pooling consistently enhances organ-tied label discrimination, with OSF providing additional gains on size- or density-driven findings; ablations show a clear ordering: GAP < Global Attention < Organ-masked Attention < Organ-masked Attention + OSF. The approach yields auditable, clinically relevant evidence maps and reduces reliance on large-scale radiology reports for supervision, offering a practical path toward calibrated, anatomy-guided CT triage in clinical deployment.

Abstract

There is an urgent need for triage and classification of high-volume medical imaging modalities such as computed tomography (CT), which can improve patient care and mitigate radiologist burnout. Study-level CT triage requires calibrated predictions with localized evidence; however, off-the-shelf Vision Language Models (VLM) struggle with 3D anatomy, protocol shifts, and noisy report supervision. This study used the two largest publicly available chest CT datasets: CT-RATE and RADCHEST-CT (held-out external test set). Our carefully tuned supervised baseline (instantiated as a simple Global Average Pooling head) establishes a new supervised state of the art, surpassing all reported linear-probe VLMs. Building on this baseline, we present ORACLE-CT, an encoder-agnostic, organ-aware head that pairs Organ-Masked Attention (mask-restricted, per-organ pooling that yields spatial evidence) with Organ-Scalar Fusion (lightweight fusion of normalized volume and mean-HU cues). In the chest setting, ORACLE-CT masked attention model achieves AUROC 0.86 on CT-RATE; in the abdomen setting, on MERLIN (30 findings), our supervised baseline exceeds a reproduced zero-shot VLM baseline obtained by running publicly released weights through our pipeline, and adding masked attention plus scalar fusion further improves performance to AUROC 0.85. Together, these results deliver state-of-the-art supervised classification performance across both chest and abdomen CT under a unified evaluation protocol. The source code is available at https://github.com/lavsendahal/oracle-ct.

Organ-Aware Attention Improves CT Triage and Classification

TL;DR

ORACLE-CT tackles the challenge of calibrated CT triage by introducing organ-aware pooling that restricts evidence to anatomically meaningful regions, paired with optional Organ-Scalar Fusion to inject concise organ-level cues. The method is encoder-agnostic and evaluated under a fixed, auditable protocol on chest (CT-RATE, RAD-ChestCT) and abdomen (MERLIN) datasets, achieving state-of-the-art supervised performance and improved calibration compared with prior VLM-based and baseline approaches. Across chest and abdomen, organ-masked pooling consistently enhances organ-tied label discrimination, with OSF providing additional gains on size- or density-driven findings; ablations show a clear ordering: GAP < Global Attention < Organ-masked Attention < Organ-masked Attention + OSF. The approach yields auditable, clinically relevant evidence maps and reduces reliance on large-scale radiology reports for supervision, offering a practical path toward calibrated, anatomy-guided CT triage in clinical deployment.

Abstract

There is an urgent need for triage and classification of high-volume medical imaging modalities such as computed tomography (CT), which can improve patient care and mitigate radiologist burnout. Study-level CT triage requires calibrated predictions with localized evidence; however, off-the-shelf Vision Language Models (VLM) struggle with 3D anatomy, protocol shifts, and noisy report supervision. This study used the two largest publicly available chest CT datasets: CT-RATE and RADCHEST-CT (held-out external test set). Our carefully tuned supervised baseline (instantiated as a simple Global Average Pooling head) establishes a new supervised state of the art, surpassing all reported linear-probe VLMs. Building on this baseline, we present ORACLE-CT, an encoder-agnostic, organ-aware head that pairs Organ-Masked Attention (mask-restricted, per-organ pooling that yields spatial evidence) with Organ-Scalar Fusion (lightweight fusion of normalized volume and mean-HU cues). In the chest setting, ORACLE-CT masked attention model achieves AUROC 0.86 on CT-RATE; in the abdomen setting, on MERLIN (30 findings), our supervised baseline exceeds a reproduced zero-shot VLM baseline obtained by running publicly released weights through our pipeline, and adding masked attention plus scalar fusion further improves performance to AUROC 0.85. Together, these results deliver state-of-the-art supervised classification performance across both chest and abdomen CT under a unified evaluation protocol. The source code is available at https://github.com/lavsendahal/oracle-ct.
Paper Structure (34 sections, 25 equations, 13 figures, 12 tables)

This paper contains 34 sections, 25 equations, 13 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Method: ORACLE– CT
  5. Problem Setup
  6. Encoders as Feature Adapters
  7. From Feature Lattices to Aggregation Heads
  8. Mask-free Heads
  9. Global Average Pooling (GAP).
  10. Global Attention (mask-free, content-weighted).
  11. Organ-aware Heads
  12. Implementation details.
  13. Training Objective and Calibration
  14. Experiments and Results
  15. Datasets and Splits
  16. ...and 19 more sections

Figures (13)

  • Figure 1: ORACLE--CT overview. Two encoder families feed a shared organ-aware head: Family A 2.5D token towers (DINOv3, MedSigLIP) and Family B native 3D trunks (CT-NET, I3D-ResNet-121, MedNeXt-3D). The head supports four pooling modes: GAP, Global attention (mask-free unary softmax over the lattice), Masked attention (softmax restricted to organ masks; one head per organ group plus an other head), and Masked + OSF (append organ volume/HU/border scalars). Outputs are study-level predictions. Organ merges and label groups are in Appendix \ref{['tab:a1-ctrate-merges', 'tab:a2-merlin-merges']}.
  • Figure 2: MERLIN (abdomen), DINOv3: per-class AUROC across four aggregation modes. We compare GAP, Global attention (mask-free), Organ-masked attention, and Masked+OSF (masked attention + organ scalars: mean HU, volume). Shown are the 25 mask-eligible classes (the 5 labels without stable organ masks are omitted). Classes are sorted by the absolute AUROC gain of Masked+OSF over GAP. Masked+OSF provides the most consistent improvements—especially for size/morphology–driven findings at the left—while organ-masked attention alone already lifts most organ-tied labels.
  • Figure 3: Qualitative organ--masked attention on MERLIN. Columns show four study examples: Atelectasis (TP), Pleural effusion (TP), Renal cyst (FP), and Renal cyst (TP). Top row: axial CT slice. Bottom row: organ mask (cyan) with masked--attention heatmap (magma). Maps depict organ-level pooling weights used for study-level classification (not lesion segmentation): they highlight where evidence within the organ contributed most to the logit. The FP renal-cyst case illustrates organ-faithful but misleading evidence.
  • Figure 4: Prevalence comparison on the harmonized 16-label chest space. Bars show per-label prevalence in CT--RATE (Internal Test) and RAD--ChestCT (External Test), ordered by CT--RATE prevalence. For visualization, CT--RATE Calcification is computed as the union of Arterial wall calcification and Coronary artery wall calcification, and Mosaic attenuation pattern is excluded because it is not annotated in RAD--ChestCT.
  • Figure 5: MERLIN (abdomen) class prevalence. Horizontal stacked bar plot showing, for each of the 30 findings (y--axis), the total number of studies (x--axis) partitioned into negatives (0), positives (1), and uncertain/missing ($-1$). Counts are aggregated over MERLIN train+val+test.
  • ...and 8 more figures