SAM3-DMS: Decoupled Memory Selection for Multi-target Video Segmentation of SAM3

TL;DR

This work tackles identity drift in multi-target video segmentation by addressing SAM3's group-level memory selection, which can incorrectly pollute memories when some targets disappear or are occluded. It introduces SAM3-DMS, a training-free decoupled memory-selection strategy that updates each target's memory independently using $S_{i,t} = q_{i,t} p_t$, preserving target-specific representations without increasing memory overhead. Empirical results across PCS and PVS benchmarks show consistent improvements, with larger gains as target density rises, demonstrating enhanced identity preservation and tracking stability in challenging open-world scenarios. The approach offers a practical and scalable enhancement to SAM3, enabling more reliable simultaneous segmentation and grounding of multiple targets in real-world videos.

Abstract

Segment Anything 3 (SAM3) has established a powerful foundation that robustly detects, segments, and tracks specified targets in videos. However, in its original implementation, its group-level collective memory selection is suboptimal for complex multi-object scenarios, as it employs a synchronized decision across all concurrent targets conditioned on their average performance, often overlooking individual reliability. To this end, we propose SAM3-DMS, a training-free decoupled strategy that utilizes fine-grained memory selection on individual objects. Experiments demonstrate that our approach achieves robust identity preservation and tracking stability. Notably, our advantage becomes more pronounced with increased target density, establishing a solid foundation for simultaneous multi-target video segmentation in the wild.

Figures (7)

• Figure 1: SAM3 vs. SAM3-DMS (Ours), in simultaneous multi-object video segmentation. In Frame #3, SAM3 updates the memories for all objects together based on the overall status, saving Object 1 into memory even its mask is blank, causing identity drifts. In contrast, we separately update memory for each object by its own, keeping consistent identity tracking.
• Figure 2: Overview of the Decoupled Memory Selection (DMS) mechanism. In SAM3, the memory status of the group is determined collectively by the average score ($Avg$), causing out-of-view objects to be "Wrongly Saved" when other group members remain visible. Our SAM3-DMS evaluates each target's status independently. This decoupling prevents corrupted features from entering the memory bank, leading to the correct identity preservation of Object 2 seen in the final frame.
• Figure 3: Example results of heavy occlusions of targets. (Case 1, zoom in for better view.)
• Figure 4: Example results of disappearance and re-appearance of targets. (Case 2, zoom in for better view.)
• Figure 5: Example results of object interferences, where distractors exhibit parallel motion. (Case 3, zoom in for better view.)