CVT-Bench: Counterfactual Viewpoint Transformations Reveal Unstable Spatial Representations in Multimodal LLMs

Abstract

Multimodal large language models (MLLMs) achieve strong performance on single-view spatial reasoning tasks, yet it remains unclear whether they maintain stable spatial state representations under counterfactual viewpoint changes. We introduce a controlled diagnostic benchmark that evaluates relational consistency under hypothetical camera orbit transformations without re-rendering images. Across 100 synthetic scenes and 6,000 relational queries, we measure viewpoint consistency, 360° cycle agreement, and relational stability over sequential transformations. Despite high single-view accuracy, state-of-the-art MLLMs exhibit systematic degradation under counterfactual viewpoint changes, with frequent violations of cycle consistency and rapid decay in relational stability. We further evaluate multiple input representations, visual input, textual bounding boxes, and structured scene graphs, and show that increasing representational structure improves stability. Our results suggest that single-view spatial accuracy overestimates the robustness of induced spatial representations and that representation structure plays a critical role in counterfactual spatial reasoning.

Figures (10)

• Figure 1: Counterfactual Viewpoint Reasoning. An agent observes spatial relations from a single viewpoint (left). When queried from a human located $180^\circ$ away, the model must transform these relations under the viewpoint change rather than reuse the original description. Although the sphere is front-right of the cube from the agent’s view, from the human’s viewpoint it becomes front-left. CVT-Bench evaluates whether multimodal models can perform such viewpoint-conditioned spatial reasoning reliably.
• Figure 2: High-level performance overview across ten diagnostic dimensions: episodic F1, long-context F1, rotation resilience ($\Delta$ F1 between original and rotated views), text benefit ($\Delta$ Text$-$Image F1), scene graph benefit ($\Delta$ SG$-$Text F1), episodic and sequential spatial relation tracking, relational consistency ($0^\circ$–$360^\circ$ match rate), and context stability ($\Delta$ F1 between episodic and sequential evaluation). Each cell displays a colored indicator: $\checkmark$ (strong), $\sim$ (moderate), or $\times$ (weak). GPT-5.2 uses BS=10.
• Figure 3: Global F1 across viewpoint rotations for episodic (top) and sequential (bottom) evaluation under Image, Text-only, and Scene Graph inputs. The anchor viewpoint ($0^\circ$) reflects direct spatial observation, while rotated viewpoints require counterfactual perspective transformation. Episodic performance is strong at $0^\circ$, but systematically degrades at intermediate rotations and declines further under sequential interaction. GPT-5.2 uses BS=10.
• Figure 4: Survival rate decay as a function of minimum consecutive-correct threshold $t$ for peak survival (top) and survival from start (bottom), across Image, Text-only, and Scene Graph modalities. Solid lines indicate BS=1 (episodic); dashed lines indicate BS=20 (sequential). The monotonic decline quantifies how rapidly spatial reasoning coherence degrades as stricter consistency requirements are imposed. GPT-5.2 uses BS=10.
• Figure 5: Performance and sequential stability across input representations. Top: Global F1 decay across batch positions. Bottom: $0^\circ{=}360^\circ$ consistency match rate drift across prompt depth, with shaded regions showing $\pm1$ standard deviation. While models achieve strong episodic performance, spatial consistency degrades under sequential interaction, indicating instability in maintaining spatial state. GPT-5.2 uses BS=10.