Specialty-Oriented Generalist Medical AI for Chest CT Screening

TL;DR

The paper tackles the challenge of building a specialty-oriented generalist medical AI capable of integrating multimodal clinical data for multiple chest CT screening tasks. It introduces M3FM, a multimodal multitask foundation model that unifies volumetric CT data with free-text clinical information via a multimodal question-answering framework, powered by CTViT and a text transformer. Through OpenM3Chest—a large, curated dataset spanning 49 data types and 17 tasks—the approach achieves state-of-the-art performance across prediction and diagnostic tasks, demonstrates solid generalization on independent datasets, and shows adaptability to out-of-distribution tasks via transfer learning. The study highlights the potential clinical impact of integrated multimodal reasoning in radiology and cardiovascular risk assessment, while acknowledging retrospective limitations and the need for scalable data pipelines and real-world validation.

Abstract

Modern medical records include a vast amount of multimodal free text clinical data and imaging data from radiology, cardiology, and digital pathology. Fully mining such big data requires multitasking; otherwise, occult but important aspects may be overlooked, adversely affecting clinical management and population healthcare. Despite remarkable successes of AI in individual tasks with single-modal data, the progress in developing generalist medical AI remains relatively slow to combine multimodal data for multitasks because of the dual challenges of data curation and model architecture. The data challenge involves querying and curating multimodal structured and unstructured text, alphanumeric, and especially 3D tomographic scans on an individual patient level for real-time decisions and on a scale to estimate population health statistics. The model challenge demands a scalable and adaptable network architecture to integrate multimodal datasets for diverse clinical tasks. Here we propose the first-of-its-kind medical multimodal-multitask foundation model (M3FM) with application in lung cancer screening and related tasks. After we curated a comprehensive multimodal multitask dataset consisting of 49 clinical data types including 163,725 chest CT series and 17 medical tasks involved in LCS, we develop a multimodal question-answering framework as a unified training and inference strategy to synergize multimodal information and perform multiple tasks via free-text prompting. M3FM consistently outperforms the state-of-the-art single-modal task-specific models, identifies multimodal data elements informative for clinical tasks and flexibly adapts to new tasks with a small out-of-distribution dataset. As a specialty-oriented generalist medical AI model, M3FM paves the way for similar breakthroughs in other areas of medicine, closing the gap between specialists and the generalist.

Figures (8)

• Figure 1: M3FM inference illustration. M3FM consists of four main components: CTViT, Text Transformer, Task Encoder, and Predictors. CTViT encodes volumetric CT images at different scales (indicated by rectangle boxes in different sizes and colors). Text Transformer encodes a combination of clinical data and questions in free text. All image and text tokens are forwarded to the Task Encoder, which extracts task-specific embedding features of the integrated multimodal data. The task-specific Predictors take task-specific embedding features to derive the final answers. These tasks may have arbitrary combinations of inputs including varied sizes of imaging data and shared/different predictors. Different colors in CT, QUESTIONS/TASKS, and ANSWERS differentiate the matches among them. The questions in the black text take the textural inputs only. Questions 17 and 18 are two examples of simulated clinical information retrieval tasks to inspect the ability of clinical data modeling. Question 19 only takes its neighboring clinical text as input.
• Figure 2: Dataset construction. (a) The general data construction workflow consists of four steps: medical tasks of interest definition, task-specific multimodal data collection, multimodal data processing and alignment, and multimodal question-answering construction; (b) The data used in this study were collected from two data centers, i.e., NLST and MIDRC, and two medical institutes, i.e., WFUSM and MGH, with the key summarized characteristics, based on which a large 3D CT pretraining dataset and a simulated clinical dataset were constructed. The green boxes indicate the OpenM3Chest dataset that will be made publicly available; (c) Distributions of main training, validation, and test datasets over all tasks are summarized; (d) Distributions of MGH independent evaluation datasets are summarized; (e) Distributions of WFUSM independent evaluation and fine-tuning datasets are summarized.
• Figure 3: Overall performance of M3FM. (a) Comparison results of the best M3FMs with previous SoTA models in terms of AUC (%), where the AUC numbers of M3FM and previous SOTA models trained under the same settings are shown and the results of other models can be found in the main text; (b) AUC (%) results with 95% CI of three scales of M3FM models. The results demonstrate that M3FM consistently surpasses previous SOTA models across all tasks. Generally, we observed that scaling up the size of the M3FM enhances its performance.
• Figure 4: Evaluation of physical size embedding in CT imaging. This figure presents the AUC (%) and 95% CI results for M3FM models with and without the embedding of CT voxel sizes across various tasks. The inclusion of physical size information significantly enhances performance in lung cancer risk prediction, cardiovascular disease (CVD) diagnosis, CVD mortality risk estimation, and nodule size characterization.
• Figure 5: Qualitative inspection of the task encoder. (a) Attention map visualization of the task encoder for two CVD diagnosis examples; (b) Attention map visualization of the task encoder for two lung cancer risk prediction examples. The exemplar results show that the task encoder has a certain ability to reveal the relevance between the model outcomes and multimodal elements. Specifically, CVD diagnosis correlates with calcification regions and the patient's history of heart disease, hypertension, diabetes, and stroke. Lung cancer risk is associated with the lung nodule region as well as demographic factors and familial lung cancer history. In row (a), the two cases were reported with significant CVD abnormalities. In row (b), the pathology test results confirmed the two patients being diagnosed with lung cancer within one year following their LDCT lung cancer screenings.