Scaling Tumor Segmentation: Best Lessons from Real and Synthetic Data

TL;DR

This work investigates how data scale affects tumor segmentation by comparing real versus real plus synthetic data using AbdomenAtlas 2.0, a large multi-organ, voxel-level annotated CT dataset. The study demonstrates an early in-distribution performance plateau with real data (around $1{,}500$ scans for pancreatic tumors) and shows that synthetic data at $3\times$ scale can achieve similar results with only $500$ real scans, thereby accelerating data efficiency. AbdomenAtlas 2.0, built with a SMART-Annotator pipeline, provides $10{,}136$ annotated scans across six organs, enabling strong in-distribution gains and substantially improved out-of-distribution generalization ($+$16\% DSC on external data). Overall, the paper offers three lessons: limit real-data annotation when possible, leverage synthetic data to steepen data-scaling curves, and prioritize data diversity to enhance cross-center robustness, with practical implications for expanding large-scale public datasets and annotation workflows in medical imaging.

Abstract

AI for tumor segmentation is limited by the lack of large, voxel-wise annotated datasets, which are hard to create and require medical experts. In our proprietary JHH dataset of 3,000 annotated pancreatic tumor scans, we found that AI performance stopped improving after 1,500 scans. With synthetic data, we reached the same performance using only 500 real scans. This finding suggests that synthetic data can steepen data scaling laws, enabling more efficient model training than real data alone. Motivated by these lessons, we created AbdomenAtlas 2.0--a dataset of 10,135 CT scans with a total of 15,130 tumor instances per-voxel manually annotated in six organs (pancreas, liver, kidney, colon, esophagus, and uterus) and 5,893 control scans. Annotated by 23 expert radiologists, it is several orders of magnitude larger than existing public tumor datasets. While we continue expanding the dataset, the current version of AbdomenAtlas 2.0 already provides a strong foundation--based on lessons from the JHH dataset--for training AI to segment tumors in six organs. It achieves notable improvements over public datasets, with a +7% DSC gain on in-distribution tests and +16% on out-of-distribution tests.

Figures (12)

• Figure 1: Data scaling laws study. Experimental results on the proprietary dataset demonstrate that increasing the scale of real data improve the segmentation (gray curve). Notably, supplementing the dataset with an additional $3\times$ synthetic data (red curve) can further enhance the results, revealing the potential of a larger public dataset to advance tumor research.
• Figure 2: Overview of the AbdomenAtlas 2.0 dataset. For each CT scan, AbdomenAtlas 2.0 provides precise and high-quality annotations following a well-designed AI-driven annotation pipeline. Compared to existing datasets, AbdomenAtlas 2.0 collects large-scale CT scans from diverse clinical sources, encompassing a wide range of tumor types (i.e., liver, pancreas, kidney, colon, esophageal, and uterine tumors) and comprehensive tumor sizes. This extensive scale makes it the largest human-annotated tumor mask dataset.
• Figure 3: Overview of the SMART-Annotator. Towards annotating a large-scale tumor dataset, developing our SMART-Annotator involves four stages. ① Train a Segmentation Model using public datasets to provide tumor segmentation logits across AbdomenAtlas 2.0. ② Analyzing the FROC curve and selecting a threshold that enhances sensitivity to minimize missed tumors while maintaining an acceptable specificity score. ③ Removing false positives for the adjusted predictions by senior radiologists. ④ Revising the final annotations to get ground truth by junior radiologists.
• Figure 4: Dataset statistics analysis on the distributions of (a) different tumor proportions, (b) tumor radius, and (c) different tumor sizes categorized as tiny, small, medium, and large.
• Figure 5: Tumor size and feature distribution of real vs. synthetic tumors. (a) Tumor size distribution across liver, pancreatic, and kidney tumors from real and synthetic data. (b–d) Feature distributions of liver, pancreatic, and kidney tumors. We extract features using a pretrained encoder rombach2022high and visualize them with t-SNE to compare synthetic tumors with real ones.