Where, Not What: Compelling Video LLMs to Learn Geometric Causality for 3D-Grounding

TL;DR

The paper addresses the persistent 2D semantic bias in multimodal 3D grounding by introducing What-Where Representation Re-Forming (W2R2), a training strategy that reshapes internal representations rather than altering the inference architecture. W2R2 employs a two-path forward pass (fused and 2D-only) with a pull--push objective, combining an alignment loss $\mathcal{L}_{\text{align}}$ and a deterrence loss $\mathcal{L}_{\text{deterrence}}$ governed by a margin $\mu$ and weight $\lambda$, to encourage 3D geometry to drive grounding while suppressing overly strong 2D shortcuts. Empirical results on ScanRefer, Scan2Cap, and ScanQA demonstrate state-of-the-art 3D grounding and competitive captioning performance, with ablations showing the importance of $\lambda$ and $\mu$ in balancing semantics and geometry. The approach preserves inference costs and offers a principled path to leverage 2D semantic capacity while achieving robust 3D spatial understanding, with potential for future symbiotic 2D–3D interactions. Key equations include $\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{align}}+\lambda\mathcal{L}_{\text{deterrence}}$ and $\mathcal{L}_{\text{deterrence}}=\max\!(0,\, s(\mathrm{IoU}_{3\mathrm{D}}(o_{\text{short}}), y)-\mu)$.

Abstract

Multimodal 3D grounding has garnered considerable interest in Vision-Language Models (VLMs) \cite{yin2025spatial} for advancing spatial reasoning in complex environments. However, these models suffer from a severe "2D semantic bias" that arises from over-reliance on 2D image features for coarse localization, largely disregarding 3D geometric inputs and resulting in suboptimal fusion performance. In this paper, we propose a novel training framework called What-Where Representation Re-Forming (W2R2) to tackle this issue via disentangled representation learning and targeted shortcut suppression. Our approach fundamentally reshapes the model's internal space by designating 2D features as semantic beacons for "What" identification and 3D features as spatial anchors for "Where" localization, enabling precise 3D grounding without modifying inference architecture. Key components include a dual-objective loss function with an Alignment Loss that supervises fused predictions using adapted cross-entropy for multimodal synergy, and a Pseudo-Label Loss that penalizes overly effective 2D-dominant pseudo-outputs via a margin-based mechanism. Experiments conducted on ScanRefer and ScanQA demonstrate the effectiveness of W2R2, with significant gains in localization accuracy and robustness, particularly in cluttered outdoor scenes.

Figures (3)

• Figure 1: t-SNE visualization of feature distributions, highlighting a 2D semantic bias. The Fused Features show a much closer proximity to the 2D Image Features than to the 3D Features.
• Figure 2: Overview of our proposed W2R2 training framework.
• Figure 3: Ablation studies for hyper-parameters $\lambda$ and $\mu$.