USF-MAE: Ultrasound Self-Supervised Foundation Model with Masked Autoencoding

TL;DR

USF-MAE tackles the scarcity of labeled ultrasound data by introducing a large-scale self-supervised Masked Autoencoding foundation model trained exclusively on ultrasound images from OpenUS-46. Using a Vision Transformer encoder-decoder, it learns modality-specific representations through masked patch reconstruction and then fine-tunes on three diverse classification tasks, achieving state-of-the-art performance among non-supervised baselines and competitive results versus supervised foundations. The results demonstrate robust cross-anatomical generalization and underline the potential of unlabeled medical data to close the gap between data availability and model performance, with public release of weights and OpenUS-46 resources enabling reproducibility and community-driven expansion. The work highlights practical clinical implications, including reduced labeling needs, a universal ultrasound feature extractor, and a path toward continual, data-efficient learning across institutions.

Abstract

Ultrasound imaging is one of the most widely used diagnostic modalities, offering real-time, radiation-free assessment across diverse clinical domains. However, interpretation of ultrasound images remains challenging due to high noise levels, operator dependence, and limited field of view, resulting in substantial inter-observer variability. Current Deep Learning approaches are hindered by the scarcity of large labeled datasets and the domain gap between general and sonographic images, which limits the transferability of models pretrained on non-medical data. To address these challenges, we introduce the Ultrasound Self-Supervised Foundation Model with Masked Autoencoding (USF-MAE), the first large-scale self-supervised MAE framework pretrained exclusively on ultrasound data. The model was pre-trained on 370,000 2D and 3D ultrasound images curated from 46 open-source datasets, collectively termed OpenUS-46, spanning over twenty anatomical regions. This curated dataset has been made publicly available to facilitate further research and reproducibility. Using a Vision Transformer encoder-decoder architecture, USF-MAE reconstructs masked image patches, enabling it to learn rich, modality-specific representations directly from unlabeled data. The pretrained encoder was fine-tuned on three public downstream classification benchmarks: BUS-BRA (breast cancer), MMOTU-2D (ovarian tumors), and GIST514-DB (gastrointestinal stromal tumors). Across all tasks, USF-MAE consistently outperformed conventional CNN and ViT baselines, achieving F1-scores of 81.6%, 79.6%, and 82.4%, respectively. Despite not using labels during pretraining, USF-MAE approached the performance of the supervised foundation model UltraSam on breast cancer classification and surpassed it on the other tasks, demonstrating strong cross-anatomical generalization.

Figures (8)

• Figure 1: OpenUS-46: A curated collection of 46 open-source US images spanning diverse anatomical regions and clinical applications.
• Figure 2: US images of two scans from AUL dataset(A) and Cactus dataset (B), and visualization of accepted vs. rejected pixels in HSV color space before inpainting (C). Accepted pixels correspond to US tissue, while rejected pixels represent annotation overlays.
• Figure 3: US image preprocessing pipelines used for annotation removal and standardization. (a-c) Cactus dataset workflow: original US image, automatically detected colored annotation mask, and the corresponding inpainted (cleaned) image using the Navier-Stokes algorithm [67]. (d-f) AUL dataset workflow: original US image, detected annotation mask using CLAHE-enhanced K-means clustering, and the inpainted result.
• Figure 4: Overview of the USF-MAE (Ultrasound Self-Supervised Foundation with Masked Autoencoding) pipeline for 1) pre-training and 2) downstream fine-tuning.
• Figure 5: US images of Benign (A) and Malignant (B) scans from the BUS-BRA dataset.