SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning

TL;DR

SSR introduces depth-aware spatial reasoning for vision-language models by translating depth into interpretable textual rationales (MIDI) and compact latent embeddings via knowledge distillation. It provides SSR-CoT, a million-scale dataset, and SSRBench, a comprehensive multi-task benchmark for assessing depth-enabled spatial reasoning. A two-stage training regime—Stage 1 alignment of MIDI with language semantics and Stage 2 joint training with a VLM—enables plug-and-play enhancement of existing VLMs. Experiments on SSRBench and SpatialBench demonstrate substantial gains in spatial understanding and depth utilization across diverse tasks, with efficiency preserved through latent reasoning. This work advances human-like multi-modal understanding while maintaining practical scalability for real-world applications.

Abstract

Despite impressive advancements in Visual-Language Models (VLMs) for multi-modal tasks, their reliance on RGB inputs limits precise spatial understanding. Existing methods for integrating spatial cues, such as point clouds or depth, either require specialized sensors or fail to effectively exploit depth information for higher-order reasoning. To this end, we propose a novel Spatial Sense and Reasoning method, dubbed SSR, a novel framework that transforms raw depth data into structured, interpretable textual rationales. These textual rationales serve as meaningful intermediate representations to significantly enhance spatial reasoning capabilities. Additionally, we leverage knowledge distillation to compress the generated rationales into compact latent embeddings, which facilitate resource-efficient and plug-and-play integration into existing VLMs without retraining. To enable comprehensive evaluation, we introduce a new dataset named SSR-CoT, a million-scale visual-language reasoning dataset enriched with intermediate spatial reasoning annotations, and present SSRBench, a comprehensive multi-task benchmark. Extensive experiments on multiple benchmarks demonstrate SSR substantially improves depth utilization and enhances spatial reasoning, thereby advancing VLMs toward more human-like multi-modal understanding. Project page: https://yliu-cs.github.io/SSR.

Figures (8)

• Figure 1: Unlike conventional VLMs, SSR integrates depth perception to enhance spatial reasoning. We introduce a curated dataset SSR-CoT and benchmark SSRBench, demonstrating significant improvements in spatial reasoning tasks.
• Figure 2: Schematic of SSR framework. (a) Overall pipeline. (b) Full architecture of SSR, comprising the MIDI module followed by the VLM. (c) Two training stages of the SSR. In the stage 1, the LLM provides alignment supervision for the MIDI module, whereas the stage 2 is optional.
• Figure 3: Schematic of SSR-CoT annotation pipeline.
• Figure 4: Illustrative samples of SSR-CoT dataset.
• Figure 5: Examples for each task within the benchmark SSRBench.