Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimation

TL;DR

The paper investigates whether Vision-Language Models (VLMs) can infer relative 3D camera motion from image pairs by evaluating them on RCPE using the VRRPI-Bench benchmark. It demonstrates that current VLMs substantially lag behind classical geometric baselines and human performance, and exhibit instability when swapping input views, indicating reliance on 2D heuristics rather than true 3D grounding. To dissect failures, the authors introduce VRRPI-Diag, which isolates single DoFs and reveals deeper difficulties with depth (z-axis) and roll transformations. Through extensive baselines, human studies, and diagnostic analyses, the work argues that robust 3D and multi-view spatial reasoning remains an open challenge for VLMs, and positions VRRPI-Bench as a rigorous stress test for progress toward grounded 3D understanding in embodied vision systems.

Abstract

Vision-Language Models (VLMs) perform well in 2D perception and semantic reasoning compared to their limited understanding of 3D spatial structure. We investigate this gap using relative camera pose estimation (RCPE), a fundamental vision task that requires inferring relative camera translation and rotation from a pair of images. We introduce VRRPI-Bench, a benchmark derived from unlabeled egocentric videos with verbalized annotations of relative camera motion, reflecting realistic scenarios with simultaneous translation and rotation around a shared object. We further propose VRRPI-Diag, a diagnostic benchmark that isolates individual motion degrees of freedom. Despite the simplicity of RCPE, most VLMs fail to generalize beyond shallow 2D heuristics, particularly for depth changes and roll transformations along the optical axis. Even state-of-the-art models such as GPT-5 ($0.64$) fall short of classic geometric baselines ($0.97$) and human performance ($0.92$). Moreover, VLMs exhibit difficulty in multi-image reasoning, with inconsistent performance (best $59.7\%$) when integrating spatial cues across frames. Our findings reveal limitations in grounding VLMs in 3D and multi-view spatial reasoning.

Figures (11)

• Figure 1: Examples from VRRPI-Bench. Camera moves from source observation (left) to target observation (right), with the verbalized camera motion for simultaneous translation and rotation around a shared object.
• Figure 2: Examples from VRRPI-Diag. Camera moves from source viewpoint (left) to target viewpoint (right) and both positive and negative descriptions are provided.
• Figure 3: Consistency performance. The dashed line denotes the random baseline ($50.0\%$), highlighting that most of VLMs are around or below random level.
• Figure 4: Results on VRRPI-Diag (top-performing VLMs of each category). The breakdown reveals a significant performance gap between image-plane shifts and optical-axis transformations. See full table in App. \ref{['app:tab-diag']}.
• Figure 5: Cross-image relational comparison results. The dashed line denotes the random baseline ($\approx 0.28$). The bars represent the "Reference Gap", the performance gain achieved when the target object is explicitly identified in the prompt. See App. \ref{['app:ea-ii']} for full results.