MV-SAM: Multi-view Promptable Segmentation using Pointmap Guidance

TL;DR

MV-SAM introduces a 3D-aware promptable segmentation framework that uses pointmaps, reconstructed from unposed images, to lift 2D prompts and image embeddings into a shared 3D space. By embedding 3D positional information and employing a lightweight transformer decoder, MV-SAM achieves view-consistent masks without explicit 3D networks or 3D supervision, and generalizes across diverse domains after training on SA-1B. The method consistently outperforms SAM2-Video and matches or approaches per-scene optimization baselines on multiple benchmarks (NVOS, SpIn-NeRF, ScanNet++, uCo3D, DL3DV), highlighting the value of 3D priors in promptable segmentation. Limitations include reliance on the quality of the underlying pointmaps and potential issues with dynamic scenes, suggesting directions for future work on uncertainty modeling and improving geometric priors.

Abstract

Promptable segmentation has emerged as a powerful paradigm in computer vision, enabling users to guide models in parsing complex scenes with prompts such as clicks, boxes, or textual cues. Recent advances, exemplified by the Segment Anything Model (SAM), have extended this paradigm to videos and multi-view images. However, the lack of 3D awareness often leads to inconsistent results, necessitating costly per-scene optimization to enforce 3D consistency. In this work, we introduce MV-SAM, a framework for multi-view segmentation that achieves 3D consistency using pointmaps -- 3D points reconstructed from unposed images by recent visual geometry models. Leveraging the pixel-point one-to-one correspondence of pointmaps, MV-SAM lifts images and prompts into 3D space, eliminating the need for explicit 3D networks or annotated 3D data. Specifically, MV-SAM extends SAM by lifting image embeddings from its pretrained encoder into 3D point embeddings, which are decoded by a transformer using cross-attention with 3D prompt embeddings. This design aligns 2D interactions with 3D geometry, enabling the model to implicitly learn consistent masks across views through 3D positional embeddings. Trained on the SA-1B dataset, our method generalizes well across domains, outperforming SAM2-Video and achieving comparable performance with per-scene optimization baselines on NVOS, SPIn-NeRF, ScanNet++, uCo3D, and DL3DV benchmarks. Code will be released.

Figures (14)

• Figure 1: MV-SAM. Our method enables view-consistent promptable segmentation, where user prompts (e.g., clicks) guide the extraction of target masks across multi-view images. The examples above illustrate visualizations of the lifted predicted masks via pointmaps, where finger points (illustrating user prompts) share the same color as the corresponding predicted masks.
• Figure 2: Overview. (a) SAM2-Video tracks masks iterative through memory modules where the visual cues are the key for tracking. In contrast, (b) our MV-SAM leverage pointmap as a unified world coordinate and embed 3D positional information in both image embeddings and user prompts to predict view-consistent masks without 3D-specific networks or large-scale 3D annotated datasets.
• Figure 3: Comparison of MV-SAM with SAM2-Video.
• Figure 4: Qualitative results on NVOS and SPIn-NeRF datasets. Compared to SAM2-Video, which often predicts incorrect regions due to the lack of 3D awareness, MV-SAM achieves consistent mask predictions. In NVOS, user prompts are provided as scribbles, where blue lines visualize positive scribbles and red lines visualize negative scribbles, whereas in SPIn-NeRF, object masks in the reference view serve as user prompts.
• Figure 5: Comparison of single-view attention (ours) and full-view attention. While both approaches perform similarly in few-view setups, full-view attention struggles to scale to a larger number of frames, which is common in practice. Detailed results are provided in the Appendix Table \ref{['table:frame_changes']}.