Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OFL-SAM2: Prompt SAM2 with Online Few-shot Learner for Efficient Medical Image Segmentation

Meng Lan, Lefei Zhang, Xiaomeng Li

TL;DR

This work addresses label-efficient medical image segmentation by removing the need for manual prompts in SAM2. It introduces OFL-SAM2, which pairs an online few-shot learner that trains a lightweight mapping network to generate discriminative target features with an adaptive fusion module that integrates these features with SAM2's memory-attention outputs, all while keeping SAM2 frozen. The model updates online during inference based on a quality-controlled selection mechanism, enabling robust performance across 3D volumes and temporally correlated 2D sequences. Experiments on Synapse-CT, PROMISE12, and Autolaparo demonstrate state-of-the-art results under limited data, highlighting its practical potential for automated, cross-modal MIS tasks.

Abstract

The Segment Anything Model 2 (SAM2) has demonstrated remarkable promptable visual segmentation capabilities in video data, showing potential for extension to medical image segmentation (MIS) tasks involving 3D volumes and temporally correlated 2D image sequences. However, adapting SAM2 to MIS presents several challenges, including the need for extensive annotated medical data for fine-tuning and high-quality manual prompts, which are both labor-intensive and require intervention from medical experts. To address these challenges, we introduce OFL-SAM2, a prompt-free SAM2 framework for label-efficient MIS. Our core idea is to leverage limited annotated samples to train a lightweight mapping network that captures medical knowledge and transforms generic image features into target features, thereby providing additional discriminative target representations for each frame and eliminating the need for manual prompts. Crucially, the mapping network supports online parameter update during inference, enhancing the model's generalization across test sequences. Technically, we introduce two key components: (1) an online few-shot learner that trains the mapping network to generate target features using limited data, and (2) an adaptive fusion module that dynamically integrates the target features with the memory-attention features generated by frozen SAM2, leading to accurate and robust target representation. Extensive experiments on three diverse MIS datasets demonstrate that OFL-SAM2 achieves state-of-the-art performance with limited training data.

OFL-SAM2: Prompt SAM2 with Online Few-shot Learner for Efficient Medical Image Segmentation

TL;DR

This work addresses label-efficient medical image segmentation by removing the need for manual prompts in SAM2. It introduces OFL-SAM2, which pairs an online few-shot learner that trains a lightweight mapping network to generate discriminative target features with an adaptive fusion module that integrates these features with SAM2's memory-attention outputs, all while keeping SAM2 frozen. The model updates online during inference based on a quality-controlled selection mechanism, enabling robust performance across 3D volumes and temporally correlated 2D sequences. Experiments on Synapse-CT, PROMISE12, and Autolaparo demonstrate state-of-the-art results under limited data, highlighting its practical potential for automated, cross-modal MIS tasks.

Abstract

The Segment Anything Model 2 (SAM2) has demonstrated remarkable promptable visual segmentation capabilities in video data, showing potential for extension to medical image segmentation (MIS) tasks involving 3D volumes and temporally correlated 2D image sequences. However, adapting SAM2 to MIS presents several challenges, including the need for extensive annotated medical data for fine-tuning and high-quality manual prompts, which are both labor-intensive and require intervention from medical experts. To address these challenges, we introduce OFL-SAM2, a prompt-free SAM2 framework for label-efficient MIS. Our core idea is to leverage limited annotated samples to train a lightweight mapping network that captures medical knowledge and transforms generic image features into target features, thereby providing additional discriminative target representations for each frame and eliminating the need for manual prompts. Crucially, the mapping network supports online parameter update during inference, enhancing the model's generalization across test sequences. Technically, we introduce two key components: (1) an online few-shot learner that trains the mapping network to generate target features using limited data, and (2) an adaptive fusion module that dynamically integrates the target features with the memory-attention features generated by frozen SAM2, leading to accurate and robust target representation. Extensive experiments on three diverse MIS datasets demonstrate that OFL-SAM2 achieves state-of-the-art performance with limited training data.
Paper Structure (21 sections, 4 equations, 3 figures, 6 tables)

This paper contains 21 sections, 4 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SAM-based Medical Image Segmentation
  4. Online Discriminative Learning
  5. Method
  6. Overall Pipeline
  7. Memory Attention module
  8. Few-shot Learner
  9. Adaptive Fusion Module
  10. Update Strategy
  11. Experiments
  12. Dataset and Evaluation
  13. Implementation details
  14. Comparison with State-of-the-art Methods
  15. PROMISE12.
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Comparisons of (a) SAM2 model and (b) our OFL-SAM2 model. SAM2 is susceptible to adjacent distractors.
  • Figure 2: An overview of our OFL-SAM2 framework.
  • Figure 3: Visualization comparison between H-SAM and OFL-SAM2 on Synapse-CT (left) and Autolaparo (right) datasets.