Synthetic Volumetric Data Generation Enables Zero-Shot Generalization of Foundation Models in 3D Medical Image Segmentation

TL;DR

This work tackles the gap between natural-image-focused foundation models and medical imaging by introducing SynthFM-3D, an analytic framework that models 3D anatomical variability, contrast, boundary definition, and noise to produce synthetic volumetric data for training promptable segmentation models. By training SAM 2.1 on 10,000 synthetic volumes and evaluating across CT, MR, and ultrasound, the approach achieves strong zero-shot generalization, outperforming the pretrained baseline and rivaling or exceeding supervised models like SAM-Med3D on unseen data. The results demonstrate that carefully parameterized synthetic data can bridge modality gaps, enabling scalable, annotation-free training of foundation models for diverse medical imaging tasks. This data-centric strategy holds practical impact for modality-agnostic segmentation, reducing reliance on manual labeling and expanding applicability to low-annotation or new-imaging-domain scenarios.

Abstract

Foundation models such as Segment Anything Model 2 (SAM 2) exhibit strong generalization on natural images and videos but perform poorly on medical data due to differences in appearance statistics, imaging physics, and three-dimensional structure. To address this gap, we introduce SynthFM-3D, an analytical framework that mathematically models 3D variability in anatomy, contrast, boundary definition, and noise to generate synthetic data for training promptable segmentation models without real annotations. We fine-tuned SAM 2 on 10,000 SynthFM-3D volumes and evaluated it on eleven anatomical structures across three medical imaging modalities (CT, MR, ultrasound) from five public datasets. SynthFM-3D training led to consistent and statistically significant Dice score improvements over the pretrained SAM 2 baseline, demonstrating stronger zero-shot generalization across modalities. When compared with the supervised SAM-Med3D model on unseen cardiac ultrasound data, SynthFM-3D achieved 2-3x higher Dice scores, establishing analytical 3D data modeling as an effective pathway to modality-agnostic medical segmentation.

Figures (2)

• Figure 1: Overview of the SynthFM-3D data generation process. Top row illustrates the analytical mapping $\Phi(M, \theta_{\text{struct}}, \theta_{\text{app}})$, where a 2D seed mask $M$ is extrapolated into a 3D label volume $Y = \Psi(M, \theta_{\text{struct}})$ and rendered into a corresponding image volume $X = \Gamma(Y, \theta_{\text{app}})$. Bottom row shows the diversity from multiple texture and contrast variations generated from the same underlying mask, showing the controllable variability achieved through different parameters $(\theta_{\text{struct}}^{(i)}, \theta_{\text{app}}^{(j)})$.
• Figure 2: Qualitative comparison of segmentation propagation between SAM 2 and SynthFM-3D. Each row corresponds to a distinct modality and two examples per modality are shown. For each case, the first column depicts the prompt slice, where the positive prompt location is marked by a green star ($\color{green}{\star}$); subsequent columns show segmentation propagation across slices.