CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoning

TL;DR

The paper tackles the challenge of robust 3D scene understanding with large multimodal models by introducing explicit high-level spatial structure. It proposes CVP, a framework that combines a central-vision-like target-affinity token with a peripheral-vision-like allocentric grid, trained via a contrastive objective and integrated textual BEV context. Built on a pre-trained LMM, CVP achieves state-of-the-art results across five benchmarks for grounding, QA, and dense captioning, with ablations confirming the complementary benefits of its two modules. Qualitative analyses and visualizations corroborate improved attention grounding and spatial reasoning in complex 3D environments.

Abstract

We present a central-peripheral vision-inspired framework (CVP), a simple yet effective multimodal model for spatial reasoning that draws inspiration from the two types of human visual fields -- central vision and peripheral vision. Existing approaches primarily rely on unstructured representations, such as point clouds, voxels, or patch features, and inject scene context implicitly via coordinate embeddings. However, this often results in limited spatial reasoning capabilities due to the lack of explicit, high-level structural understanding. To address this limitation, we introduce two complementary components into a Large Multimodal Model-based architecture: target-affinity token, analogous to central vision, that guides the model's attention toward query-relevant objects; and allocentric grid, akin to peripheral vision, that captures global scene context and spatial arrangements. These components work in tandem to enable structured, context-aware understanding of complex 3D environments. Experiments show that CVP achieves state-of-the-art performance across a range of 3D scene understanding benchmarks.

Figures (4)

• Figure 1: Overview of CVP, inspired by human visual cognition. (a) Human vision combines central vision (dark blue) for focused, high-acuity perception and peripheral vision (light blue) for broader contextual awareness. (b) Our model mimics this dual process with target-affinity token, guiding attention to target-relevant objects/regions, and allocentric grid, capturing allocentric spatial context. (c) Quantitative results across multiple 3D scene understanding benchmarks. We report EM on SQA3D sqa3d, CIDEr on ScanQA scanqa and Scan2Cap scan2cap, Acc@0.25 on ScanRefer scanrefer, and F1@0.25 on Multi3DRefer multi3drefer. With target-affinity token and allocentric allocentric grid, CVP consistently outperforms state-of-the-art models such as LLaVA-3D llava3d and Video-3D-LLM video3dllm.
• Figure 2: Illustration of CVP. Given visual tokens from multi-view images with 3D positional embeddings and a user question as input, we (1) incorporate a text-based allocentric grid to provide allocentric global scene context; and (2) introduce a special target-affinity token that guides the model to focus on target-related objects. During output generation, in addition to producing a language response, the representation of the target-affinity token is passed through an MLP and optimized with a contrastive loss against 3D object embeddings back-projected from multi-view 2D features. Positive samples correspond to ground-truth objects relevant to the question, while negatives are irrelevant. This contrastive supervision helps the model attend more effectively to semantically relevant targets.
• Figure 3: Object retrieval results using our target-affinity token. We compute similarities between the output target-affinity embedding and all object embeddings in the scene. The top-1 result is shown in a green box; top-3 results are highlighted in orange.
• Figure 4: Qualitative comparisons across three representative 3D vision-language tasks: dense captioning, visual grounding, and question answering. CVP consistently outperforms the baseline model.