Towards Better Ultrasound Video Segmentation Foundation Model: An Empirical study on SAM2 Finetuning from Data Perspective

TL;DR

This study investigates how to adapt Segment Anything Model 2 to ultrasound video segmentation from a data-centric perspective. It systematically compares task-specific fine-tuning, midtraining, and multi-task joint training across multiple SAM2 variants, prompting modes, and ultrasound-specific augmentations, revealing that training data scale and temporal context often trump architectural changes. Key findings show that data quantity and temporal context significantly influence performance, short videos boost learning mainly in data-scarce regimes, and spatial augmentations grounded in imaging physics yield more reliable gains than generic methods. The results offer practical guidance for data-aware SAM2 adaptation pipelines in ultrasound video analysis, emphasizing data scale and domain-aligned augmentations over mere model scaling. These insights have potential to improve efficiency and generalizability in clinical ultrasound segmentation workflows.

Abstract

Ultrasound (US) video segmentation remains a challenging problem due to strong inter- and intra-dataset variability, motion artifacts, and limited annotated data. Although foundation models such as Segment Anything Model 2 (SAM2) demonstrate strong zero-shot and prompt-guided segmentation capabilities, their performance deteriorates substantially when transferred to medical imaging domains. Current adaptation studies mainly emphasize architectural modifications, while the influence of data characteristics and training regimes has not been systematically examined. In this study, we present a comprehensive, data-centric investigation of SAM2 adaptation for ultrasound video segmentation. We analyze how training-set size, video duration, and augmentation schemes affect adaptation performance under three paradigms: task-specific fine-tuning, intermediate adaptation, and multi-task joint training, across five SAM2 variants and multiple prompting modes. We further design six ultrasound-specific augmentations, assessing their effect relative to generic strategies. Experiments on three representative ultrasound datasets reveal that data scale and temporal context play a more decisive role than model architecture or initialization. Moreover, joint training offers an efficient compromise between modality alignment and task specialization. This work aims to provide empirical insights for developing efficient, data-aware adaptation pipelines for SAM2 in ultrasound video analysis.

Figures (11)

• Figure 1: Illustration of data variation in medical ultrasound video datasets. The figure highlights both inter-dataset and intra-dataset variability, including domain shifts across datasets, class imbalance, and differences in video length, temporal resolution, and content diversity within a single dataset. These factors can significantly influence fine-tuning performance and model generalization in medical video-based machine learning.
• Figure 2: Dataset overview and partition scheme. Left: overview of the three core target datasets, showing frame counts for short and long videos. Right: illustration of the training–testing split, construction of multi-scale training subsets, and generation of the synthetic balanced short-video training set.
• Figure 3: Illustration of adaptation regimes in this study. (a) Task-specific fine-tuning (FT): each model is fine-tuned and tested on a specific core target dataset (with training and testing dataset split). (b) Midtraining (MT) and midtraining–fine-tuning (MT-FT): for each Core Target Dataset, MT is performed on all Auxiliary and the rest of Core Target Datasets. During MT-FT, Gray dashed lines indicate the corresponding training Core Target Datasets for each testing one. (c) Joint Training (JT): model is trained on all Core Target Datasets simultaneously.
• Figure 4: Visualization of the spatial augmentation strategies. The first row (purple) shows augmentations proposed in this study, and the second row (gray) shows the original image augmentations along with the SAM2-default. Dash-lined boxes highlight structural changes in the augmented results.
• Figure 5: Visualization of the temporal augmentation strategies. The first and last row (gray) shows the SAM2-default augmentations along with the original video. The second to fourth row (purple) shows augmentations proposed in this study. a, b, c denote frame IDs highlighting changes in frame sequence.