Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models

TL;DR

Actial addresses the gap between 2D visual understanding and robust 3D spatial reasoning in Multimodal LLMs by introducing Viewpoint Learning and the Viewpoint-100K dataset. It deploys a two-stage fine-tuning pipeline—foundational knowledge injection via supervised fine-tuning on viewpoint tasks with a hybrid cold-start, followed by generalization enhancement through GRPO on the SAT spatial dataset—to activate 3D spatial reasoning and improve cross-view consistency. Experimental results on 3DSRBench, CV-Bench, and BLINK show significant gains in in-domain and out-of-domain spatial tasks, with ablations highlighting the importance of both SFT and GRPO components and the potential trade-offs. The work offers a practical pathway to improve 3D perception in robotics and autonomous systems, while acknowledging limitations such as dataset scope and the need for explicit 3D grounding beyond object-centric settings.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have significantly improved 2D visual understanding, prompting interest in their application to complex 3D reasoning tasks. However, it remains unclear whether these models can effectively capture the detailed spatial information required for robust real-world performance, especially cross-view consistency, a key requirement for accurate 3D reasoning. Considering this issue, we introduce Viewpoint Learning, a task designed to evaluate and improve the spatial reasoning capabilities of MLLMs. We present the Viewpoint-100K dataset, consisting of 100K object-centric image pairs with diverse viewpoints and corresponding question-answer pairs. Our approach employs a two-stage fine-tuning strategy: first, foundational knowledge is injected to the baseline MLLM via Supervised Fine-Tuning (SFT) on Viewpoint-100K, resulting in significant improvements across multiple tasks; second, generalization is enhanced through Reinforcement Learning using the Group Relative Policy Optimization (GRPO) algorithm on a broader set of questions. Additionally, we introduce a hybrid cold-start initialization method designed to simultaneously learn viewpoint representations and maintain coherent reasoning thinking. Experimental results show that our approach significantly activates the spatial reasoning ability of MLLM, improving performance on both in-domain and out-of-domain reasoning tasks. Our findings highlight the value of developing foundational spatial skills in MLLMs, supporting future progress in robotics, autonomous systems, and 3D scene understanding.

Figures (8)

• Figure 1: We aim to activate the MLLM's spatial reasoning ability with Viewpoint Learning and the two-stage fine-tuning strategy.
• Figure 2: 2D Continuity and 3D Consistency. 2D continuity refers to the high similarity between adjacent frames, whereas 3D consistency focuses on preserving spatial and geometric relationships across frames. Top: Verifying 3D consistency requires estimating the camera pose and comparing these spatial properties in 3D space. Bottom: Adjusting the scale of each video frame slightly can destroy 3D consistency while maintaining 2D continuity.
• Figure 3: Overview of our pipeline. We introduce Actial, which comprises a novel dataset and a two-stage fine-tuning strategy. In the knowledge injection phase, we employ a hybrid cold-start initialization to enhance the model's foundational spatial skills and leverage pseudo CoTs to ensure robust reasoning capabilities. Subsequently, we enhance the model's generalization capabilities through a specialized generalization enhancement stage.
• Figure 4: Thoughts on viewpoint question. Current MLLMs tend to rely on 2D cues to address viewpoint-related problems, which often leads to incorrect reasoning and erroneous results.
• Figure 5: The reasoning process. Actial uses the correct spatial thinking approach.