Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning

Byungwoo Jeon, Dongyoung Kim, Huiwon Jang, Insoo Kim, Jinwoo Shin

Abstract

Despite the remarkable success of large-scale pre-trained image representation models (i.e., vision encoders) across various vision tasks, they are predominantly trained on 2D image data and therefore often fail to capture 3D spatial relationships between objects and backgrounds in the real world, constraining their effectiveness in many downstream applications. To address this, we propose SpatialBoost, a scalable framework that enhances the spatial awareness of existing pre-trained vision encoders by injecting 3D spatial knowledge expressed in linguistic descriptions. The core idea involves converting dense 3D spatial information from 2D images into linguistic expressions, which is then used to inject such spatial knowledge into vision encoders through a Large Language Model (LLM). To this end, we adopt a multi-turn Chain-of-Thought (CoT) reasoning process that progressively incorporates dense spatial knowledge and builds hierarchical spatial understanding. To validate effectiveness, we adapt SpatialBoost to state-of-the-art vision encoders such as DINOv3, and evaluate its performance gains on a wide range of benchmarks requiring both 3D perception and general vision abilities. For instance, SpatialBoost improves DINOv3 performance from 55.9 to 59.7 mIoU on ADE20K, achieving state-of-the-art performance with 3.8% gain over the pre-trained DINOv3.

SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning

Abstract

Despite the remarkable success of large-scale pre-trained image representation models (i.e., vision encoders) across various vision tasks, they are predominantly trained on 2D image data and therefore often fail to capture 3D spatial relationships between objects and backgrounds in the real world, constraining their effectiveness in many downstream applications. To address this, we propose SpatialBoost, a scalable framework that enhances the spatial awareness of existing pre-trained vision encoders by injecting 3D spatial knowledge expressed in linguistic descriptions. The core idea involves converting dense 3D spatial information from 2D images into linguistic expressions, which is then used to inject such spatial knowledge into vision encoders through a Large Language Model (LLM). To this end, we adopt a multi-turn Chain-of-Thought (CoT) reasoning process that progressively incorporates dense spatial knowledge and builds hierarchical spatial understanding. To validate effectiveness, we adapt SpatialBoost to state-of-the-art vision encoders such as DINOv3, and evaluate its performance gains on a wide range of benchmarks requiring both 3D perception and general vision abilities. For instance, SpatialBoost improves DINOv3 performance from 55.9 to 59.7 mIoU on ADE20K, achieving state-of-the-art performance with 3.8% gain over the pre-trained DINOv3.
Paper Structure (35 sections, 3 equations, 8 figures, 21 tables)

This paper contains 35 sections, 3 equations, 8 figures, 21 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Self-supervised Learning for Image Representation
  4. Multi-modal Learning for Image Representation
  5. Multi-View Learning for Image Representation
  6. Method
  7. Training Pipeline
  8. Enhancing Vision Encoder with Spatial CoT
  9. Experiments
  10. Experimental Setup
  11. Dense Prediction Tasks
  12. Complex 3D-centric Tasks
  13. Vision-based Robot Learning
  14. Image Classification and Retrieval Tasks
  15. Ablation Study and Analysis
  16. ...and 20 more sections

Figures (8)

  • Figure 1: Overview of SpatialBoost. We enhance spatial and geometric understanding of pre-trained vision encoders by leveraging language-guided spatial reasoning. SpatialBoost consists of (a) spatial knowledge extraction through depth estimation, 3D reconstruction, segmentation, and region captioning, (b) converting spatial knowledge into multi-turn spatial reasoning from pixel to scene levels, and (c) building a spatial-aware vision encoder with LLM using generated data in (b).
  • Figure 1: Results on monocular depth estimation from NYUd silberman2012indoor and KITTI geiger2013vision benchmarks. We report the RMSE score between ground truth and predicted depth values. Lower is better. For all results, we freeze the encoder backbone and train a linear head (lin.) or DPT head ranftl2021vision on top of the image features of the last layer.
  • Figure 2: Illustration of multi-turn visual spatial reasoning dataset, exhibiting pixel-level, object-level, and scene-level reasoning QAs. At the pixel-level, the QA task queries the 3D positions of points (, via depth estimation). At the object-level, it extracts spatial properties of objects (, by predicting bounding cubes or relative positions). At the scene-level, it determines the exact distances between multiple objects that require the rationales of the previous steps. At last, we add 2-turn for general scene caption. These are listed in order and constitute 12 multi-turn visual spatial reasoning conservation.
  • Figure 3: Illustration of the dual-channel attention layerhong2022cogvideo, where an additional attention block is introduced alongside the original attention block and merged via a learnable mixture factor $\alpha$.
  • Figure 4: Results on vision-based robot learning. We report the performance of imitation learning agents on 4 domains from CortexBench majumdar2023we, which are trained upon the image representations. In particular, we report the normalized score for DMControl and success rates (%) for other tasks.
  • ...and 3 more figures