Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoSAM: Motion-Guided Segment Anything Model with Spatial-Temporal Memory Selection

Qiushi Yang, Yuan Yao, Miaomiao Cui, Liefeng Bo

TL;DR

MoSAM addresses two core limitations of SAM2 in video segmentation: lack of explicit motion information and unreliable memory from past frames. It introduces Motion-Guided Prompting (MGP) to inject sparse and dense motion cues as prompts for subsequent frames, and Spatial-Temporal Memory Selection (ST-MS) to curate a high-quality memory bank at both temporal and spatial levels. Through extensive experiments on VOS, VIS, and OpenVIS benchmarks, MoSAM achieves state-of-the-art performance and demonstrates robust handling of occlusion and object disappearance, with ablations confirming the complementary benefits of MGP and ST-MS. The approach offers practical impact for reliable long-term object tracking in open-world video understanding tasks.

Abstract

The recent Segment Anything Model 2 (SAM2) has demonstrated exceptional capabilities in interactive object segmentation for both images and videos. However, as a foundational model on interactive segmentation, SAM2 performs segmentation directly based on mask memory from the past six frames, leading to two significant challenges. Firstly, during inference in videos, objects may disappear since SAM2 relies solely on memory without accounting for object motion information, which limits its long-range object tracking capabilities. Secondly, its memory is constructed from fixed past frames, making it susceptible to challenges associated with object disappearance or occlusion, due to potentially inaccurate segmentation results in memory. To address these problems, we present MoSAM, incorporating two key strategies to integrate object motion cues into the model and establish more reliable feature memory. Firstly, we propose Motion-Guided Prompting (MGP), which represents the object motion in both sparse and dense manners, then injects them into SAM2 through a set of motion-guided prompts. MGP enables the model to adjust its focus towards the direction of motion, thereby enhancing the object tracking capabilities. Furthermore, acknowledging that past segmentation results may be inaccurate, we devise a Spatial-Temporal Memory Selection (ST-MS) mechanism that dynamically identifies frames likely to contain accurate segmentation in both pixel- and frame-level. By eliminating potentially inaccurate mask predictions from memory, we can leverage more reliable memory features to exploit similar regions for improving segmentation results. Extensive experiments on various benchmarks of video object segmentation and video instance segmentation demonstrate that our MoSAM achieves state-of-the-art results compared to other competitors.

MoSAM: Motion-Guided Segment Anything Model with Spatial-Temporal Memory Selection

TL;DR

MoSAM addresses two core limitations of SAM2 in video segmentation: lack of explicit motion information and unreliable memory from past frames. It introduces Motion-Guided Prompting (MGP) to inject sparse and dense motion cues as prompts for subsequent frames, and Spatial-Temporal Memory Selection (ST-MS) to curate a high-quality memory bank at both temporal and spatial levels. Through extensive experiments on VOS, VIS, and OpenVIS benchmarks, MoSAM achieves state-of-the-art performance and demonstrates robust handling of occlusion and object disappearance, with ablations confirming the complementary benefits of MGP and ST-MS. The approach offers practical impact for reliable long-term object tracking in open-world video understanding tasks.

Abstract

The recent Segment Anything Model 2 (SAM2) has demonstrated exceptional capabilities in interactive object segmentation for both images and videos. However, as a foundational model on interactive segmentation, SAM2 performs segmentation directly based on mask memory from the past six frames, leading to two significant challenges. Firstly, during inference in videos, objects may disappear since SAM2 relies solely on memory without accounting for object motion information, which limits its long-range object tracking capabilities. Secondly, its memory is constructed from fixed past frames, making it susceptible to challenges associated with object disappearance or occlusion, due to potentially inaccurate segmentation results in memory. To address these problems, we present MoSAM, incorporating two key strategies to integrate object motion cues into the model and establish more reliable feature memory. Firstly, we propose Motion-Guided Prompting (MGP), which represents the object motion in both sparse and dense manners, then injects them into SAM2 through a set of motion-guided prompts. MGP enables the model to adjust its focus towards the direction of motion, thereby enhancing the object tracking capabilities. Furthermore, acknowledging that past segmentation results may be inaccurate, we devise a Spatial-Temporal Memory Selection (ST-MS) mechanism that dynamically identifies frames likely to contain accurate segmentation in both pixel- and frame-level. By eliminating potentially inaccurate mask predictions from memory, we can leverage more reliable memory features to exploit similar regions for improving segmentation results. Extensive experiments on various benchmarks of video object segmentation and video instance segmentation demonstrate that our MoSAM achieves state-of-the-art results compared to other competitors.
Paper Structure (21 sections, 4 figures, 4 tables)

This paper contains 21 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Object Segmentation
  4. Segment Anything Model
  5. Methodology
  6. Preliminaries: SAM2
  7. Motion-Guided Prompting (MGP)
  8. Sparse Motion Modeling
  9. Dense Motion Modeling
  10. Spatial-Temporal Memory Selection (ST-MS)
  11. Temporal Memory Selection
  12. Spatial Memory Selection
  13. Experiments
  14. Datasets and Details
  15. Comparison with SAM2
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Overview of the proposed MoSAM framework. It consists of a motion-guided prompting (MGP) mechanism to inject object motion cues into the model for superior object tracking, and a spatial-temporal memory selection (ST-MS) strategy, dynamically updating feature memory to maintain a reliable and effective memory bank.
  • Figure 2: Illustration of the proposed spatial-temporal memory selection (ST-MS) strategy, which can pick up relative confident frame features in the temporal level and the reliable pixels of each frame in the spatial level to update the memory bank.
  • Figure 3: Qualitative comparison on video object segmentation. (a), (c) show the results from SAM2, and (b),(d) are drawn from our MoSAM, superior in hard cases including object object disappearance and occlusion. Red boxes suggest the wrong segmentation or object object disappearance, and green boxes indicate accurate segmentation.
  • Figure 4: Analysis on the hyper-parameters of our MoSAM framework. (a) Number of sparse motion cues as key points; (b) Time interval for optical flow; (c) Thresholds of IoU and Occlusion scores; (d) Threshold of frames to be ranked.