Accelerating Volumetric Medical Image Annotation via Short-Long Memory SAM 2

TL;DR

SLM-SAM 2 extends SAM 2 with a dual memory architecture—a long-term memory bank for stable propagation and a short-term memory bank for rapid adaptation—coupled with a dynamic attention fuser to improve segmentation across 3D medical volumes. By reading current features from both memories and fusing them, the model significantly improves Dice scores and reduces boundary errors during slice propagation, while also substantially lowering the manual effort required to correct propagated masks. The approach demonstrates robust gains across organ, bone, muscle, and fetal head segmentation in MRI, CT, and ultrasound datasets, and it integrates into 3D Slicer to support practical annotation workflows. Overall, SLM-SAM 2 provides a practical, scalable method for faster, more accurate volumetric medical image annotation with meaningful time savings for clinical and research use.

Abstract

Manual annotation of volumetric medical images, such as magnetic resonance imaging (MRI) and computed tomography (CT), is a labor-intensive and time-consuming process. Recent advancements in foundation models for video object segmentation, such as Segment Anything Model 2 (SAM 2), offer a potential opportunity to significantly speed up the annotation process by manually annotating one or a few slices and then propagating target masks across the entire volume. However, the performance of SAM 2 in this context varies. Our experiments show that relying on a single memory bank and attention module is prone to error propagation, particularly at boundary regions where the target is present in the previous slice but absent in the current one. To address this problem, we propose Short-Long Memory SAM 2 (SLM-SAM 2), a novel architecture that integrates distinct short-term and long-term memory banks with separate attention modules to improve segmentation accuracy. We evaluate SLM-SAM 2 on four public datasets covering organs, bones, and muscles across MRI, CT, and ultrasound videos. We show that the proposed method markedly outperforms the default SAM 2, achieving an average Dice Similarity Coefficient improvement of 0.14 and 0.10 in the scenarios when 5 volumes and 1 volume are available for the initial adaptation, respectively. SLM-SAM 2 also exhibits stronger resistance to over-propagation, reducing the time required to correct propagated masks by 60.575% per volume compared to SAM 2, making a notable step toward more accurate automated annotation of medical images for segmentation model development.

Figures (13)

• Figure 1: Overview of SLM-SAM 2. (a) is the general pipeline of SLM-SAM 2. The dynamic short-long memory module consists of short-term and long-term memory banks ($\mathcal{M}_{short}$ and $\mathcal{M}_{long}$), two distinct attention modules for each of them ($\mathcal{A}_{short}$ and $\mathcal{A}_{long}$), and the dynamic attention fuser ($\mathcal{F}$). Specifically, $\mathcal{M}_{long}$ includes memories from one annotated slice and up to six additional slices, while $\mathcal{M}_{short}$ contains memory only from the most recent slice. $A_{short}$ and $A_{long}$ denote the outputs of $\mathcal{A}_{short}$ and $\mathcal{A}_{long}$, respectively, and $A_{SLM-SAM 2}$ represents the fused attention features produced by $\mathcal{F}$. (b) illustrates the architecture of the dynamic attention fuser $\mathcal{F}$. (c) provides an explanation of the key notations used in SLM-SAM 2.
• Figure 2: Average performance comparison across datasets (5-Volume). SLM-SAM 2 demonstrates the best average performance compared among all baselines in both DSC and ASSD.
• Figure 3: Performance results (DSC) of each method across datasets (5-Volume). SLM-SAM 2 outperforms all baselines on most datasets by a significant margin. Confidence intervals are estimated using bootstrapping with 1000 resamples.
• Figure 4: Qualitative comparison on each dataset (5-Volume). The leftmost column shows the annotated slices, followed by the ground truth (GT) and segmentations from each method. For each dataset, we present results on both target-present and target-absent slices.
• Figure 5: Performance results on target-present and target-absent slices (5-Volume). (a) presents 2D slice-level DSC for both target-present slices and target-absent slices across datasets. (b) plots 2D slice-level DSC over slice index on an MRI-Kidney example. (c)-(e) provide segmentation comparison between SAM 2 and SLM-SAM 2 on representative slices from the same MRI-Kidney volume.