X2CT-CLIP: Enable Multi-Abnormality Detection in Computed Tomography from Chest Radiography via Tri-Modal Contrastive Learning

TL;DR

CT screening is limited by radiation, cost, and turnaround, while CXR is safer but cannot easily detect multiple CT-diagnosed abnormalities. The authors introduce X2CT-CLIP, a tri-modal contrastive learning framework that aligns chest radiographs with CT volumes and CT reports in a shared latent space, using simulated CXRs from CT data and freezing the CT-CLIP backbone to train only the CXR encoder. The approach, governed by the objective L_X2CT(C,R,X) = α L(C,R) + β L(X,R) + γ L(X,C) with α=0, β=γ=1, yields superior cross-modal retrieval, few-shot adaptation, and external validation across CT-RATE, RadChest-CT, and MIMIC-CT. This enables CT-level disease prediction from CXR with reduced computational requirements, offering a scalable 2D-to-3D transfer pathway for resource-limited clinical settings and broader studies in multi-modal medical imaging.

Abstract

Computed tomography (CT) is a key imaging modality for diagnosis, yet its clinical utility is marred by high radiation exposure and long turnaround times, restricting its use for larger-scale screening. Although chest radiography (CXR) is more accessible and safer, existing CXR foundation models focus primarily on detecting diseases that are readily visible on the CXR. Recently, works have explored training disease classification models on simulated CXRs, but they remain limited to recognizing a single disease type from CT. CT foundation models have also emerged with significantly improved detection of pathologies in CT. However, the generalized application of CT-derived labels on CXR has remained illusive. In this study, we propose X2CT-CLIP, a tri-modal knowledge transfer learning framework that bridges the modality gap between CT and CXR while reducing the computational burden of model training. Our approach is the first work to enable multi-abnormality classification in CT, using CXR, by transferring knowledge from 3D CT volumes and associated radiology reports to a CXR encoder via a carefully designed tri-modal alignment mechanism in latent space. Extensive evaluations on three multi-label CT datasets demonstrate that our method outperforms state-of-the-art baselines in cross-modal retrieval, few-shot adaptation, and external validation. These results highlight the potential of CXR, enriched with knowledge derived from CT, as a viable efficient alternative for disease detection in resource-limited settings.

Figures (1)

• Figure 1: Left: Tri-modal contrastive learning framework of the CT, CT report, and CXR triplet. Right: Latent space alignment of CT, CT report, and CXR features.