Scaling Self-Supervised and Cross-Modal Pretraining for Volumetric CT Transformers

TL;DR

SPECTRE tackles the challenge of scaling self-supervised and cross-modal pretraining for volumetric CT by introducing a geometry-aware, two-branch 3D Vision Transformer with minimal 3D tokenization and 3D Rotary Positional Encoding. It employs a two-stage pretraining regime—DINOv3-like self-supervision for local features, followed by SigLIP cross-modal alignment with radiology reports—to learn representations that are both geometrically faithful and clinically meaningful. Across biomarker classification, semantic segmentation, and zero-shot CT–report retrieval, SPECTRE demonstrates strong zero-shot and transfer performance, outperforming prior CT foundation models on several benchmarks and providing an open, scalable 3D CT backbone. The work highlights a scalable direction for open-source, fully transformer-based 3D medical foundations that can support diverse clinical tasks while mitigating reliance on private data.

Abstract

We introduce SPECTRE, a fully transformer-based foundation model for volumetric computed tomography (CT). Our Self-Supervised & Cross-Modal Pretraining for CT Representation Extraction (SPECTRE) approach utilizes scalable 3D Vision Transformer architectures and modern self-supervised and vision-language pretraining strategies to learn general-purpose CT representations. Volumetric CT poses unique challenges, such as extreme token scaling, geometric anisotropy, and weak or noisy clinical supervision, that make standard transformer and contrastive learning recipes ineffective out of the box. The framework jointly optimizes a local transformer for high-resolution volumetric feature extraction and a global transformer for whole-scan context modeling, making large-scale 3D attention computationally tractable. Notably, SPECTRE is trained exclusively on openly available CT datasets, demonstrating that high-performing, generalizable representations can be achieved without relying on private data. Pretraining combines DINO-style self-distillation with SigLIP-based vision-language alignment using paired radiology reports, yielding features that are both geometrically consistent and clinically meaningful. Across multiple CT benchmarks, SPECTRE consistently outperforms prior CT foundation models in both zero-shot and fine-tuned settings, establishing SPECTRE as a scalable, open, and fully transformer-based foundation model for 3D medical imaging.

Figures (7)

• Figure 1: Radar plot comparing 11 CT foundation models across six biomarker classification benchmarks using frozen-embedding kNN classifiers. Diagnostic tasks on chest CT are shown in orange, prognostic tasks on chest CT in green, and prognostic tasks on abdominal CT in blue. SPECTRE achieves the highest performance on four of the six benchmarks, demonstrating stronger and more transferable volumetric representations compared to prior models.
• Figure 2: Overview of the proposed multimodal CT–report model. The model jointly processes volumetric CT data and corresponding radiology reports. The local vision transformer ViT$_{\ell}$, pretrained using DINOv3 (Stage 1), extracts localized image features from CT volume crops. These features are aggregated by the global vision transformer ViT$_g$, while the text transformer encodes the associated medical report. During SigLIP pretraining (Stage 2), the vision and text representations are aligned in a shared embedding space.
• Figure 3: Quantitative comparison of 11 CT foundation models across six biomarker classification benchmarks using frozen-embedding kNN classifiers. Bars represent mean performance for each task, with error bars indicating 95% confidence intervals across cross-validation folds.
• Figure 4: Non-curated saliency maps of SPECTRE on six tumor image biomarker datasets, obtained by occlusion sensitivity.
• Figure 5: SEoMT architecture, derived from the EoMT. A learnable query for each class C is initialized and concatenated to the patch tokens. The new set of tokens are jointly processed by the last $L_2$ blocks and used to predict logits corresponding to the semantic masks.