Prompt-Free SAM-Based Multi-Task Framework for Breast Ultrasound Lesion Segmentation and Classification

TL;DR

This work addresses the challenge of simultaneous segmentation and classification of breast ultrasound lesions by introducing a prompt-free SAM-based multi-task framework. It leverages a SAM vision encoder to extract rich embeddings, decoded by lightweight or UNet-like heads for segmentation, while segmentation-derived attention guides a classifier for three-way diagnosis. The model is trained with a joint loss $\mathcal{L}_{\text{total}} = \lambda \mathcal{L}_{\text{seg}} + (1-\lambda) \mathcal{L}_{\text{cls}}$ where $\lambda = 0.6$, achieving a Dice coefficient of $0.887$ and an accuracy of $92.3\%$ (top entries on PRECISE 2025) and demonstrating that SAM-based representations coupled with segmentation-guided learning improve both lesion delineation and diagnostic prediction. Overall, the approach offers an efficient, end-to-end framework for automated breast ultrasound analysis with strong clinical relevance and potential for extension to multi-view or multi-modal imaging.

Abstract

Accurate tumor segmentation and classification in breast ultrasound (BUS) imaging remain challenging due to low contrast, speckle noise, and diverse lesion morphology. This study presents a multi-task deep learning framework that jointly performs lesion segmentation and diagnostic classification using embeddings from the Segment Anything Model (SAM) vision encoder. Unlike prompt-based SAM variants, our approach employs a prompt-free, fully supervised adaptation where high-dimensional SAM features are decoded through either a lightweight convolutional head or a UNet-inspired decoder for pixel-wise segmentation. The classification branch is enhanced via mask-guided attention, allowing the model to focus on lesion-relevant features while suppressing background artifacts. Experiments on the PRECISE 2025 breast ultrasound dataset, split per class into 80 percent training and 20 percent testing, show that the proposed method achieves a Dice Similarity Coefficient (DSC) of 0.887 and an accuracy of 92.3 percent, ranking among the top entries on the PRECISE challenge leaderboard. These results demonstrate that SAM-based representations, when coupled with segmentation-guided learning, significantly improve both lesion delineation and diagnostic prediction in breast ultrasound imaging.

Figures (1)

• Figure 1: Overview of our proposed multi-task learning framework. The image encoder extracts shared features, which are processed by a mask decoder for segmentation and an attention-enhanced module for classification into benign, malignant, or normal categories.