Developing a PET/CT Foundation Model for Cross-Modal Anatomical and Functional Imaging

TL;DR

This work addresses the lack of foundation models for multimodal PET/CT in oncology by introducing Cross-Fraternal Twin Masked Autoencoder (FratMAE), a dual-encoder, cross-attention framework that learns joint anatomical, functional, and textual representations. By processing whole-body coronal patch stacks and aligning PET features with radiotracer metadata through a ContextAlign contrastive objective, FratMAE achieves strong performance on downstream tasks such as lesion segmentation and Hodgkin lymphoma staging, especially in data-limited scenarios. Experimental results show that coronal Patch-based training outperforms axial approaches and that FratMAE surpasses various pretraining strategies, highlighting its data efficiency and generalization potential. The proposed model paves the way for scalable, cross-modal PET/CT analysis with broader clinical impact, including future radiotracer treatment-response predictions and modality-specific extensions.

Abstract

In oncology, Positron Emission Tomography-Computed Tomography (PET/CT) is widely used in cancer diagnosis, staging, and treatment monitoring, as it combines anatomical details from CT with functional metabolic activity and molecular marker expression information from PET. However, existing artificial intelligence-driven PET/CT analyses rely predominantly on task-specific models trained from scratch or on limited datasets, limiting their generalizability and robustness. To address this, we propose a foundation model approach specifically designed for multimodal PET/CT imaging. We introduce the Cross-Fraternal Twin Masked Autoencoder (FratMAE), a novel framework that effectively integrates whole-body anatomical and functional or molecular information. FratMAE employs separate Vision Transformer (ViT) encoders for PET and CT scans, along with cross-attention decoders that enable synergistic interactions between modalities during masked autoencoder training. Additionally, it incorporates textual metadata to enhance PET representation learning. By pre-training on PET/CT datasets, FratMAE captures intricate cross-modal relationships and global uptake patterns, achieving superior performance on downstream tasks and demonstrating its potential as a generalizable foundation model.

Figures (2)

• Figure 1: (A) Traditional 3D medical image foundation model training. (B) Proposed FratMAE training leveraging coronal scan stacks to capture comprehensive metabolic activity. (C) Downstream tasks include segmentation and clinical staging.
• Figure 2: Visual comparison of lesion segmentation using different pre-training methods.