SCALE-VLP: Soft-Weighted Contrastive Volumetric Vision-Language Pre-training with Spatial-Knowledge Semantics

TL;DR

SCALE-VLP addresses the core challenges of applying vision-language pre-training to volumetric medical data by introducing Soft-Weighted Contrastive Alignment (SWCA) that combines continuous semantic affinity with volumetric spatial coherence and medical knowledge priors. The framework uses a dual-encoder architecture (3D ViT for CT volumes and BioClinicalBERT for reports) and a soft, pairwise loss that accounts for intra-modal similarities, cross-modal relevance, and domain knowledge, all guided by spatial and knowledge-based weights. It demonstrates strong cross-task transfer (CT–report retrieval, report generation, abnormality classification) and cross-domain generalization (zero-shot on BIMCV-R), outperforming state-of-the-art methods and showing robust, clinically meaningful alignments through ablations and qualitative analyses. The work offers practical impact by enabling efficient, clinically grounded understanding of 3D radiology data with limited supervision, improving diagnostic support and information retrieval without extensive labeled data.

Abstract

Vision-language models (VLMs) have demonstrated strong cross-modal capabilities, yet most work remains limited to 2D data and assumes binary supervision (i.e., positive vs. negative pairs), overlooking the continuous and structured dependencies present in volumetric data such as CT. Existing approaches often treat volumetric scans as independent 2D slices, compromising spatial coherence and underutilizing rich clinical semantics. We propose SCALE-VLP, a soft-weighted contrastive vision-language pre-training framework that integrates (i) volumetric spatial semantics to preserve anatomical structure and (ii) domain-aware, knowledge-infused semantics (e.g., radiological ontologies) to guide alignment. This yields structurally consistent and semantically grounded representations under limited supervision, demonstrating strong cross-task transferability (retrieval, report generation, and classification), and cross-domain generalizability with consistent gains without further fine-tuning. In particular, compared to the previous state of the art, SCALE-VLP achieves up to 4.3x higher top-1 CT-report retrieval, improves abnormality classification by 10 points, and reaches ROUGE-L 0.44 and BERT-F1 0.89 for report generation. Further, in zero-shot evaluation on an out-of-domain external dataset, we observe consistent gains, indicating the cross-task and cross-domain generalization ability of SCALE-VLP.

Figures (8)

• Figure 1: SCALE-VLP framework. A 3D vision encoder embeds CT volumes and a clinical-text encoder embeds reports. Soft-Weighted Contrastive Alignment aligns modalities using feature similarity, spatial proximity, and medical knowledge priors.
• Figure 2: Batch scaling effect
• Figure 3: Similarity Matrices by Model and Cluster. Panels (b–c) share axes: x = report embeddings; y = CT scan embeddings.
• Figure 4: Framework of multi‑task SCALE-VLP for clinical downstream tasks: (1) report generation, (2) CT–report retrieval, and (3) CT abnormality classification.
• Figure 5: Effect of the mixing parameter $\alpha$ on CT $\rightarrow$ Report retrieval.