A Spatial-Temporal Progressive Fusion Network for Breast Lesion Segmentation in Ultrasound Videos

TL;DR

This work tackles the challenge of breast lesion segmentation in ultrasound videos, where boundary blur, noise, and temporal variability hinder reliable delineation. It introduces STPFNet, a Spatial-Temporal Progressive Fusion Network that fuses intra-frame and inter-frame cues through a Temporal Fusion Module, a Spatial Fusion Module, and a Multi-Scale Feature Fusion module, while leveraging the segmentation of the previous frame as prior knowledge. A new UVBLS200 dataset comprising 200 ultrasound video sequences (80 benign, 120 malignant) is released to enable robust evaluation in realistic video settings. Experimental results show that STPFNet achieves superior performance over state-of-the-art methods, with notable improvements in Dice, IoU, Recall, and MAE, and qualitative analyses demonstrate sharper boundaries and better noise suppression. The work advances ultrasound video lesion segmentation by integrating spatial-temporal cues and prior-frame information, with potential to improve early detection and treatment planning in clinical practice.

Abstract

Ultrasound video-based breast lesion segmentation provides a valuable assistance in early breast lesion detection and treatment. However, existing works mainly focus on lesion segmentation based on ultrasound breast images which usually can not be adapted well to obtain desirable results on ultrasound videos. The main challenge for ultrasound video-based breast lesion segmentation is how to exploit the lesion cues of both intra-frame and inter-frame simultaneously. To address this problem, we propose a novel Spatial-Temporal Progressive Fusion Network (STPFNet) for video based breast lesion segmentation problem. The main aspects of the proposed STPFNet are threefold. First, we propose to adopt a unified network architecture to capture both spatial dependences within each ultrasound frame and temporal correlations between different frames together for ultrasound data representation. Second, we propose a new fusion module, termed Multi-Scale Feature Fusion (MSFF), to fuse spatial and temporal cues together for lesion detection. MSFF can help to determine the boundary contour of lesion region to overcome the issue of lesion boundary blurring. Third, we propose to exploit the segmentation result of previous frame as the prior knowledge to suppress the noisy background and learn more robust representation. In particular, we introduce a new publicly available ultrasound video breast lesion segmentation dataset, termed UVBLS200, which is specifically dedicated to breast lesion segmentation. It contains 200 videos, including 80 videos of benign lesions and 120 videos of malignant lesions. Experiments on the proposed dataset demonstrate that the proposed STPFNet achieves better breast lesion detection performance than state-of-the-art methods.

Figures (6)

• Figure 1: In the figure the first two rows represent benign lesions and their ground truth, and the last two rows represent malignant lesions and their ground truth. From left to right are the three types of challenges in our dataset, which are the significant change in shape of the lesions, the blurring of the boundaries, and the presence of regions similar to the lesions, respectively.
• Figure 2: In our proposed network, we denote frames 1 to t-1 in the video as past frames. The term "pred-t" denotes the multiplication of the prediction mask from the previous frame with the current frame. "t-1" represents the previous frame. Subsequently, the "Pred-t" and the previous frame "t-1" are fed into the network backbone, thus generating $\mathbf{V}^{p}_{t-1}$ and $\mathbf{K}^{p}_{t-1}$ respectively. These obtained features are then fused together by the MSFF module, and the final prediction is produced by the decoder.
• Figure 3: This is our spatial fusion module. The prediction of the previous frame ($H_{o}\times W_{o}\times 1$) is multiplied with the current frame ($H_{o}\times W_{o}\times C_{o}$) as a priori knowledge to suppress the background noise.
• Figure 4: The multi-scale feature fusion module is proposed. Features obtained by spatial-temporal fusion are categorised as fine-grained information, while features obtained with convolution only are categorised as coarse-grained information. The features are fused using a weighted summation performed by two Linear layers.
• Figure 5: Visual presentation of comparative experiments. The first column corresponds to the ground truth, the second column shows the baseline prediction results, the third column shows the AFB prediction results, the fourth row shows the DCF prediction results, the fifth row shows the UFO prediction results, and the sixth column represents the prediction results of our method. Where the breast lesion area is highlighted in red.