SpatialGeo:Boosting Spatial Reasoning in Multimodal LLMs via Geometry-Semantics Fusion

TL;DR

SpatialGeo addresses the limited spatial reasoning of multimodal LLMs by fusing geometry-aware MoGe features with CLIP's semantic representations in a hierarchical adapter. The approach reveals and mitigates spatial ambiguity in CLIP embeddings, enabling a more grounded spatial understanding when processed by the LLM. Through a two-stage training regime and random feature dropping, SpatialGeo achieves at least an 8% improvement on SpatialRGPT-Bench with about 50% lower memory usage, while maintaining performance on general VQA tasks. This work offers a practical path to enhanced 3D spatial reasoning in vision-language models for applications in robotics and embodied AI, with efficient training and inference.

Abstract

Multimodal large language models (MLLMs) have achieved significant progress in image and language tasks due to the strong reasoning capability of large language models (LLMs). Nevertheless, most MLLMs suffer from limited spatial reasoning ability to interpret and infer spatial arrangements in three-dimensional space. In this work, we propose a novel vision encoder based on hierarchical fusion of geometry and semantics features, generating spatial-aware visual embedding and boosting the spatial grounding capability of MLLMs. Specifically, we first unveil that the spatial ambiguity shortcoming stems from the lossy embedding of the vision encoder utilized in most existing MLLMs (e.g., CLIP), restricted to instance-level semantic features. This motivates us to complement CLIP with the geometry features from vision-only self-supervised learning via a hierarchical adapter, enhancing the spatial awareness in the proposed SpatialGeo. The network is efficiently trained using pretrained LLaVA model and optimized with random feature dropping to avoid trivial solutions relying solely on the CLIP encoder. Experimental results show that SpatialGeo improves the accuracy in spatial reasoning tasks, enhancing state-of-the-art models by at least 8.0% in SpatialRGPT-Bench with approximately 50% less memory cost during inference. The source code is available via https://ricky-plus.github.io/SpatialGeoPages/.

Figures (3)

• Figure 1: SpatialGeo demonstrates superior performance in complex spatial reasoning tasks compared to SOTA models, including open-source LLaVA-v1.5 llava2024, closed-source GPT-4.1 gpt4.1 and 3D-LLM SpatialRGPT cheng2025spatialrgpt.
• Figure 2: Left: The gap between CLIP and MoGe embedding space, where CLIP fails to distinguish images with different spatial information. Right: Overall architecture of SpatialGeo composed of CLIP module, MoGe module with hierarchical adapter, and LLM module with mixture of geometry and semantic features. The hierarchical adapter is first optimized with freezed vision encoders; then in instruction tuning, all adapters and LLM are updated with a random feature dropping strategy.
• Figure 3: Sample VQA in SpatialRGPT-Bench cheng2025spatialrgpt. Compared to LLaVA-v1.5-7B llava2024, GPT-4.1 gpt4.1 and SpatialRGPT cheng2025spatialrgpt, SpatialGeo exhibits superior performance in quantitative spatial VQA tasks, e.g., inferring object sizes and relative distances.