TotalFM: An Organ-Separated Framework for 3D-CT Vision Foundation Models

TL;DR

TotalFM addresses the computational bottlenecks of 3D-CT foundation models by introducing an organ-separated learning framework that builds organ-level volume-text representations. It automates data construction with TotalSegmentator and LLM processing, pretrains with VideoMAE on a large unlabeled CT corpus, and applies organ-wise contrastive learning before coupling to an LLM via Q-Former and LoRA for VLM fine-tuning. In extensive zero-shot evaluations, TotalFM achieves superior organ-wise F1 and finding-wise AUROC compared with CT-CLIP and Merlin, and delivers radiology report generation performance comparable to existing VLMs. The approach offers a practical, scalable pathway to deploy high-resolution 3D-CT foundation models in clinical settings and to support downstream diagnostic and reporting tasks.

Abstract

While foundation models in radiology are expected to be applied to various clinical tasks, computational cost constraints remain a major challenge when training on 3D-CT volumetric data. In this study, we propose TotalFM, a radiological foundation model that efficiently learns the correspondence between 3D-CT images and linguistic expressions based on the concept of organ separation, utilizing a large-scale dataset of 140,000 series. By automating the creation of organ volume and finding-sentence pairs through segmentation techniques and Large Language Model (LLM)-based radiology report processing, and by combining self-supervised pre-training via VideoMAE with contrastive learning using volume-text pairs, we aimed to balance computational efficiency and representation capability. In zero-shot organ-wise lesion classification tasks, the proposed model achieved higher F1 scores in 83% (5/6) of organs compared to CT-CLIP and 64% (9/14) of organs compared to Merlin. These results suggest that the proposed model exhibits high generalization performance in a clinical evaluation setting using actual radiology report sentences. Furthermore, in zero-shot finding-wise lesion classification tasks, our model achieved a higher AUROC in 83% (25/30) of finding categories compared to Merlin. We also confirmed performance comparable to existing Vision-Language Models (VLMs) in radiology report generation tasks. Our results demonstrate that the organ-separated learning framework can serve as a realistic and effective design guideline for the practical implementation of 3D-CT foundation models.

Figures (5)

• Figure 1: Overview of the proposed organ–separated framework for 3D–CT vision foundation models. The proposed framework consists of the following steps: (a) structuring CT volumes and radiology reports into organ–level units using TotalSegmentator and large language models (LLMs); (b) self–supervised pre–training of the image encoder using VideoMAE on a large–scale CT dataset (J-MID); (c) organ–wise contrastive learning between segmented CT volumes and corresponding structured report sentences; (d) evaluation via zero–shot classification tasks at both organ and finding levels; and (e) construction of a vision––language model by connecting the image encoder and an LLM using a Q–Former and LoRA.
• Figure 2: Overview of the Volume–Text Pair Construction Pipeline. DCM indicates DICOM data containing images and metadata. Seg. denotes organ segmentations generated by TotalSegmentator.
• Figure 3: Overview of VLM Fine–tuning with Organ–specific Embeddings.
• Figure 4: Comparison of AUROC performance by finding category. (a) AUROC comparison for 30 finding categories defined in the J-MID dataset. (b) AUROC comparison for 17 finding categories selected from the Merlin Test dataset.
• Figure 5: Examples of organ–specific radiology reports generated by TotalFM (VLM) and their corresponding human evaluations by a radiologist.