Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Tracking

Haofeng Liu, Mingqi Gao, Xuxiao Luo, Ziyue Wang, Guanyi Qin, Junde Wu, Yueming Jin

TL;DR

This paper tackles the challenge of real-time, long-range surgical referring segmentation driven by textual prompts. It introduces ReSurgSAM2, a two-stage framework that first performs text-referred target detection using SAM2 enhanced by CSTMamba and CIFS to identify a credible initial frame, then switches to robust tracking augmented by Diversity-Driven Long-term Memory (DLM). The approach achieves state-of-the-art accuracy with real-time performance (61.2 FPS) on Ref-EndoVis17/18 datasets, significantly outperforming both offline and online baselines and demonstrated through comprehensive ablations. By enabling interactive, memory-augmented segmentation in long surgical videos, ReSurgSAM2 supports more precise AR overlays and surgeon-guided navigation, potentially improving surgical safety and outcomes.

Abstract

Surgical scene segmentation is critical in computer-assisted surgery and is vital for enhancing surgical quality and patient outcomes. Recently, referring surgical segmentation is emerging, given its advantage of providing surgeons with an interactive experience to segment the target object. However, existing methods are limited by low efficiency and short-term tracking, hindering their applicability in complex real-world surgical scenarios. In this paper, we introduce ReSurgSAM2, a two-stage surgical referring segmentation framework that leverages Segment Anything Model 2 to perform text-referred target detection, followed by tracking with reliable initial frame identification and diversity-driven long-term memory. For the detection stage, we propose a cross-modal spatial-temporal Mamba to generate precise detection and segmentation results. Based on these results, our credible initial frame selection strategy identifies the reliable frame for the subsequent tracking. Upon selecting the initial frame, our method transitions to the tracking stage, where it incorporates a diversity-driven memory mechanism that maintains a credible and diverse memory bank, ensuring consistent long-term tracking. Extensive experiments demonstrate that ReSurgSAM2 achieves substantial improvements in accuracy and efficiency compared to existing methods, operating in real-time at 61.2 FPS. Our code and datasets will be available at https://github.com/jinlab-imvr/ReSurgSAM2.

ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Tracking

TL;DR

This paper tackles the challenge of real-time, long-range surgical referring segmentation driven by textual prompts. It introduces ReSurgSAM2, a two-stage framework that first performs text-referred target detection using SAM2 enhanced by CSTMamba and CIFS to identify a credible initial frame, then switches to robust tracking augmented by Diversity-Driven Long-term Memory (DLM). The approach achieves state-of-the-art accuracy with real-time performance (61.2 FPS) on Ref-EndoVis17/18 datasets, significantly outperforming both offline and online baselines and demonstrated through comprehensive ablations. By enabling interactive, memory-augmented segmentation in long surgical videos, ReSurgSAM2 supports more precise AR overlays and surgeon-guided navigation, potentially improving surgical safety and outcomes.

Abstract

Surgical scene segmentation is critical in computer-assisted surgery and is vital for enhancing surgical quality and patient outcomes. Recently, referring surgical segmentation is emerging, given its advantage of providing surgeons with an interactive experience to segment the target object. However, existing methods are limited by low efficiency and short-term tracking, hindering their applicability in complex real-world surgical scenarios. In this paper, we introduce ReSurgSAM2, a two-stage surgical referring segmentation framework that leverages Segment Anything Model 2 to perform text-referred target detection, followed by tracking with reliable initial frame identification and diversity-driven long-term memory. For the detection stage, we propose a cross-modal spatial-temporal Mamba to generate precise detection and segmentation results. Based on these results, our credible initial frame selection strategy identifies the reliable frame for the subsequent tracking. Upon selecting the initial frame, our method transitions to the tracking stage, where it incorporates a diversity-driven memory mechanism that maintains a credible and diverse memory bank, ensuring consistent long-term tracking. Extensive experiments demonstrate that ReSurgSAM2 achieves substantial improvements in accuracy and efficiency compared to existing methods, operating in real-time at 61.2 FPS. Our code and datasets will be available at https://github.com/jinlab-imvr/ReSurgSAM2.
Paper Structure (9 sections, 3 equations, 2 figures, 4 tables)

This paper contains 9 sections, 3 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. ReSurgSAM2 Framework
  4. Identify the Reliable Initial Frame with CSTMamba and CIFS
  5. Diversity-driven Long-term Memory
  6. Experiment
  7. Dataset and Implementation Details
  8. Comparison and Ablation Study
  9. Conclusion

Figures (2)

  • Figure 1: Overview of ReSurgSAM2. The model begins with the text-referred target detection using CSTMamba to provide credible frames for selection using CIFS. Upon detecting the initial frame, CIFS activates the tracking stage, in which DLM offers diverse and reliable memory for consistent long-term tracking.
  • Figure 2: Qualitative comparison in Ref-EndoVis17(left) and Ref-EndoVis18(right).