MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation

TL;DR

This work defines MV-3DRES, a setting where language grounding must be performed directly from sparse multi-view RGB images without dense 3D inputs. It introduces MVGGT, a dual-branch transformer with a frozen geometric reconstruction pathway and a trainable multimodal pathway that injects language cues into sparse-view reasoning through cross-view attention and geometric injection, enabling end-to-end joint reconstruction and grounding. The paper identifies Foreground Gradient Dilution (FGD) as a core optimization barrier in sparse 3D supervision and proposes Per-view No-target Suppression Optimization (PVSO), including 2D gradient concentration and targeted no-target suppression, to stabilize training. To ensure reproducible evaluation, MVRefer benchmarks MV-3DRES with standardized settings, metrics, and data protocol, plus diagnostic view-level metrics to separate grounding from reconstruction. Empirical results show MVGGT achieves strong accuracy and fast inference, outperforming existing baselines on MVRefer and standard ScanRefer protocols, and establishes a strong baseline for multimodal 3D grounding under sparse sensing conditions, with code and models available publicly.

Abstract

Most existing 3D referring expression segmentation (3DRES) methods rely on dense, high-quality point clouds, while real-world agents such as robots and mobile phones operate with only a few sparse RGB views and strict latency constraints. We introduce Multi-view 3D Referring Expression Segmentation (MV-3DRES), where the model must recover scene structure and segment the referred object directly from sparse multi-view images. Traditional two-stage pipelines, which first reconstruct a point cloud and then perform segmentation, often yield low-quality geometry, produce coarse or degraded target regions, and run slowly. We propose the Multimodal Visual Geometry Grounded Transformer (MVGGT), an efficient end-to-end framework that integrates language information into sparse-view geometric reasoning through a dual-branch design. Training in this setting exposes a critical optimization barrier, termed Foreground Gradient Dilution (FGD), where sparse 3D signals lead to weak supervision. To resolve this, we introduce Per-view No-target Suppression Optimization (PVSO), which provides stronger and more balanced gradients across views, enabling stable and efficient learning. To support consistent evaluation, we build MVRefer, a benchmark that defines standardized settings and metrics for MV-3DRES. Experiments show that MVGGT establishes the first strong baseline and achieves both high accuracy and fast inference, outperforming existing alternatives. Code and models are publicly available at https://mvggt.github.io.

Figures (4)

• Figure 1: The Reality Gap: From Idealized 3D RES to Real-World MV-3DRES.(a) Traditional 3D RES depends on dense, high-quality point clouds produced by slow offline scanning and heavy reconstruction. (b) Applied to sparse, low-quality point clouds from real-world RGB views, these models fail to generalize. (c) We introduce MV-3DRES, which uses sparse multi-view RGB inputs and text to achieve robust joint reconstruction and perception, enabled by our MVGGT model.
• Figure 2: Illustration of Foreground Gradient Dilusion Problem of Global 3D DICE loss and Per-view No-Target Suppression Optimization.
• Figure 3: Architecture of MVGGT, which comprises a frozen Reconstruction Branch that establishes geometric structure and a trainable Multimodal Branch that integrates language into sparse-view visual reasoning.
• Figure 4: Qualitative comparison on the MVRefer benchmark.