Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Masks

TL;DR

This work tackles the high cost of manual tumor masks for semantic CT segmentation by introducing R-Super, a training framework that learns from radiology reports (CT–Report pairs) to segment tumors across multiple organs. It leverages three novel losses—Volume Loss, Ball Loss, and Attenuation Loss—together with zero-shot LLM-extracted tumor characteristics to supervise segmentation without requiring exhaustive masks, while still benefiting from masks when available. Trained on 101{,}654 CT–Report pairs (plus 723 radiologist-corrected masks) and validated across internal and external datasets, R-Super achieves comparable performance to mask-based training and surpasses radiologist and public vision–language models in several tumor types, including small lesions. The results demonstrate scalable, cross-organ tumor segmentation, enabling opportunistic early detection in routine CT workflows and offering a data-and-methods framework to democratize multi-tumor AI segmentation with minimal manual annotation.

Abstract

Early tumor detection save lives. Each year, more than 300 million computed tomography (CT) scans are performed worldwide, offering a vast opportunity for effective cancer screening. However, detecting small or early-stage tumors on these CT scans remains challenging, even for experts. Artificial intelligence (AI) models can assist by highlighting suspicious regions, but training such models typically requires extensive tumor masks--detailed, voxel-wise outlines of tumors manually drawn by radiologists. Drawing these masks is costly, requiring years of effort and millions of dollars. In contrast, nearly every CT scan in clinical practice is already accompanied by medical reports describing the tumor's size, number, appearance, and sometimes, pathology results--information that is rich, abundant, and often underutilized for AI training. We introduce R-Super, which trains AI to segment tumors that match their descriptions in medical reports. This approach scales AI training with large collections of readily available medical reports, substantially reducing the need for manually drawn tumor masks. When trained on 101,654 reports, AI models achieved performance comparable to those trained on 723 masks. Combining reports and masks further improved sensitivity by +13% and specificity by +8%, surpassing radiologists in detecting five of the seven tumor types. Notably, R-Super enabled segmentation of tumors in the spleen, gallbladder, prostate, bladder, uterus, and esophagus, for which no public masks or AI models previously existed. This study challenges the long-held belief that large-scale, labor-intensive tumor mask creation is indispensable, establishing a scalable and accessible path toward early detection across diverse tumor types. We plan to release our trained models, code, and dataset at https://github.com/MrGiovanni/R-Super

Figures (8)

• Figure 1: (a) CT-Report datasets are much larger than CT-Mask datasets. Our dataset has 117K CT-Report pairs, 98K with tumors. Merlin (public) blankemeier2024merlin has 25K CT-Report pairs, 16K with tumors. In contrast, the largest CT-Mask datasets have 3K CT-Mask pairs with tumors. No tumor mask is available for many tumor types in CT. The figure also shows an example CT scan with a pancreatic tumor (PDAC), part of its report, and its tumor mask (red). (b) Overview of R-Super training method. R-Super transforms reports into per-voxel supervision for tumor segmentation, through new loss functions. It can train on both CT-Mask pairs and CT-Report pairs. For CT-Mask, R-Super uses usual dice and cross-entropy segmentation losses. For CT-Report, R-Super uses the new Volume Loss (Section \ref{['sec:volume_loss']}), Ball Loss (Section \ref{['sec:ball_loss']}) and Attenuation Loss (Section \ref{['sec:att_loss']}). They optimize the segmentation output of the AI, enforcing consistency between segmented tumors and the tumor characteristics in the report---tumor count, diameters, locations, estimated volumes and attenuation. This information is extracted from the reports by an LLM and stored before training. R-Super is applicable to any segmentation architecture with minimal extra computational cost (the zero-shot LLM runs once, before training). The figure shows a spleen tumor in a CT and its segmentation by R-Super (red).
• Figure 2: Trained on 101K CT-Report pairs, R-Super segments seven understudied tumor types. For adrenal tumors, 53 public tumor masks exist moawad2023voxel. For the other six tumor types, no mask exists. By learning from reports, R-Super can segment these tumors---becoming the first public AI that segments them in CT. R-Super surpasses radiologist tumor detection performance for five of the seven tumor types. Radiologist performance was extracted from the literature, see Appendix \ref{['app:radiologist_studies']} for an analysis of the selected studies and limitations of the comparison. Training with 101K CT-Report pairs surpassed training with 723 masks, showing that large-scale weak supervision (many reports) can surpass small-scale strong supervision (few masks). Training with both the 101K CT-Report pairs and the 723 CT-Mask pairs provided $+9\%/+6\%/+10\%$ sensitivity/ specificity/F1-Score improvement over standard segmentation (no report). Here, we train on UCSF and Merlin, and test on UCSF test---N=1,220 (see dataset descriptions in \ref{['tab:datasets_summary']}). Radiologist performances in tumor detection were acquired from several studies:
• Figure 3: In external validation, R-Super outperforms standard segmentation (trained only with CT-Mask pairs) by large margins. R-Super surpasses radiologist tumor detection performance for six of the seven tumor types. Radiologist performance was extracted from the literature, see Appendix \ref{['app:radiologist_studies']} for an analysis of the selected studies and limitations of the comparison. Even when trained with CT-Report pairs (620 to 11K CT-Report pairs per tumor type) and zero CT-Mask pairs, R-Super surpassed standard segmentation (trained with 29 to 183 CT-Mask pairs per tumor type, \ref{['tab:datasets_summary']}). R-Super trained with both CT-Report pairs and CT-Mask pairs achieved the best results, surpassing standard segmentation by +12% F1-Score. We test on a hospital never seen during training, the Stanford Hospital (Merlin Test Set, N=1,133). All segmentation models were trained on the UCSF dataset and tested on Merlin. Esophagus tumor F1-Score seem low due to a large unbalance in the test set: only 21 esophagus tumor cases for 170 normals. See \ref{['tab:datasets_summary']} for dataset details.
• Figure 4: R-Super scales the largest public pancreatic tumor segmentation dataset, improving AI performance---especially for small tumors ($<$ 2 cm). PanTS li2025pants is the largest public CT-Mask dataset for pancreatic tumor segmentation (1.1K pancreatic tumor masks). We scale it by merging PanTS and Merlin blankemeier2024merlin, a public CT-Report dataset with 2K pancreatic tumor reports. By learning from CT-Mask and CT-Report pairs (PanTS & Merlin), R-Super substantially outperformed a standard segmentation model, trained only on CT-Mask pairs (PanTS). We evaluated in a Merlin test set (400 CTs, 200 with pancreatic tumors), and the PanTS test set (901 CTs, 151 with pancreatic tumors). Notably, R-Super had the largest advantage in small pancreatic tumors (e.g., +19% sensitivity for small tumors in PanTS)---critical for early detection. We evaluated both for tumor detection (sensitivity and specificity) and segmentation (DSC and NSD). DSC and NSD are only possible to calculate in PanTS, because it has ground-truth tumor masks. Also, we calculate DSC and NSD only for CT scans with tumors. For small tumors, R-Super produced a strong improvement in DSC and NSD. For larger tumors, the improvement was smaller, possibly indicating an overfit of the standard segmentation model (trained on PanTS only) to the PanTS masks.
• Figure 5: Dataset summary.(a) Distribution of tumor diameters and patient sex in the training dataset (UCSF & Merlin train) and test datasets (UCSF test and Merlin test). (b) Distribution of tumor counts per CT scan and patient age in the training and test datasets. (c) Comparison of age and sex distribution for healthy and unhealthy (tumor) patients. The UCSF test set was randomly selected, matching the age and sex distribution from healthy and unhealthy patients---avoiding bias in our results.