3AM: Segment Anything with Geometric Consistency in Videos

TL;DR

3AM introduces a training-time fusion of 3D-aware MUSt3R features with the 2D promptable SAM2 backbone to achieve geometry-consistent video object segmentation without requiring camera poses or 3D supervision at inference. A Feature Merger combines multi-level 3D representations with SAM2's appearance cues, while a field-of-view–aware training strategy ensures learning aligns with overlapping 3D regions across frames. Empirical results on ScanNet++ and Replica demonstrate substantial gains over SAM2 and its extensions, especially under large viewpoint changes and object reappearance, with 2D tracking translating into competitive 3D instance segmentation. This approach enables robust, promptable, online VOS with strong cross-view identity preservation in real-world, geometry-rich scenes.

Abstract

Video object segmentation methods like SAM2 achieve strong performance through memory-based architectures but struggle under large viewpoint changes due to reliance on appearance features. Traditional 3D instance segmentation methods address viewpoint consistency but require camera poses, depth maps, and expensive preprocessing. We introduce 3AM, a training-time enhancement that integrates 3D-aware features from MUSt3R into SAM2. Our lightweight Feature Merger fuses multi-level MUSt3R features that encode implicit geometric correspondence. Combined with SAM2's appearance features, the model achieves geometry-consistent recognition grounded in both spatial position and visual similarity. We propose a field-of-view aware sampling strategy ensuring frames observe spatially consistent object regions for reliable 3D correspondence learning. Critically, our method requires only RGB input at inference, with no camera poses or preprocessing. On challenging datasets with wide-baseline motion (ScanNet++, Replica), 3AM substantially outperforms SAM2 and extensions, achieving 90.6% IoU and 71.7% Positive IoU on ScanNet++'s Selected Subset, improving over state-of-the-art VOS methods by +15.9 and +30.4 points. Project page: https://jayisaking.github.io/3AM-Page/

Figures (14)

• Figure 1: Limitations of traditional VOS and 3D segmentation approaches, and an overview of our capability. (a) Traditional VOS methods such as SAM2 ravi2024sam2 lose track when the camera undergoes large viewpoint changes, causing masks to drift or disappear. (b) 3D segmentation approaches rely on accurate camera poses and explicit 3D mask merging; they often propagate errors when the 3D reconstruction is incomplete or noisy. (c) Our 3AM consistently tracks object instances across drastic viewpoint changes without requiring camera poses or 3D ground-truth masks, demonstrating robust cross-view correspondence purely from geometry-aware 2D tracking.
• Figure 2: 3AM Pipeline Overview. Our Feature Merger fuses multi-level MUSt3R features, learned from multi-view consistency to encode implicit geometric correspondence, with SAM2's appearance features via cross-attention and convolutional refinement. These merged geometry-aware representations then undergo memory attention with previous frames and mask decoding, enabling spatially-consistent object recognition that maintains identity across large viewpoint changes without requiring camera poses at inference.
• Figure 3: Illustration of Features for Feature Merging. The heat map is computed using the cosine similarity between the red query point and the target frame. As illustrated in the lower row, vanilla SAM2 fails under large viewpoint changes. In contrast, as the MUSt3R feature hierarchy gradually shifts from semantic correspondence toward the point-cloud domain, we select intermediate layers to preserve both semantic relevance and geometric structure. By combining MUSt3R's geometric cues with SAM2's visual semantics, the merged feature $F_{{merged}}$ provides a significantly more reliable localization of the target object.
• Figure 4: Overview of our sampling strategy during training. (a) Continuous sampling provides densely spaced frames but offers limited viewpoint diversity. (b) Naïve random sampling introduces viewpoint variation but may select frames that observe spatially disjoint regions of the same object. For example, frame 0 shows the left side of the couch while frame 1 shows the right side. Because these regions are far apart in 3D space, treating them as the same supervisory signal forces the model to match inconsistent geometry and leads to ambiguous learning. (c) Our field-of-view–aware sampling retains only frames whose masked 3D points lie within the reference camera frustum over a threshold, ensuring consistent geometric overlap while preserving natural pose and occlusion variation.
• Figure 5: Visual comparison of VOS methods. The leftmost frame is used as the conditioned frame and provides the reference mask.