Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generation

Yuchen Li, Amanmeet Garg, Shalini Chaudhuri, Rui Zhao, Garin Kessler

Abstract

Large Vision Language Models (LVLMs) excel at semantic understanding but struggle with fine grained spatial grounding, as the model must implicitly infer complex geometry without ever producing a spatial interpretation. We present Perceptio, a perception enhanced LVLM with 2D and 3D spatial reasoning abilities, enabled via explicit semantic segmentation tokens and depth tokens generated directly within the autoregressive sequence. Concretely, we (i) distill a VQVAE depth codebook from a strong monocular teacher to tokenize dense depth into compact sequences, and (ii) integrate SAM2 based semantic segmentation tokens and VQ-VAE depth tokens inside the LLM so the model first emits spatial tokens and then answers. To stabilize depth token generation, we introduce novel composite depth-token objectives (marker, token, and count losses) and a soft-merging technique for differentiable reconstruction. We adopt a multi-task co-training strategy across diverse datasets, letting the model learn perception tokens to tackle multiple downstream tasks. Building on InternVL, Perceptio achieves state-of-the-art performance across benchmarks: improving referring expression segmentation by +0.8/+1.4/+1.1 cIoU on RefCOCO/+/g HardBLINK spatial understanding accuracy by 10.3%, and MMBench accuracy by 1.0%, demonstrating that explicit spatial chain-of-thought materially strengthens spatial grounding in LVLMs.

Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generation

Abstract

Large Vision Language Models (LVLMs) excel at semantic understanding but struggle with fine grained spatial grounding, as the model must implicitly infer complex geometry without ever producing a spatial interpretation. We present Perceptio, a perception enhanced LVLM with 2D and 3D spatial reasoning abilities, enabled via explicit semantic segmentation tokens and depth tokens generated directly within the autoregressive sequence. Concretely, we (i) distill a VQVAE depth codebook from a strong monocular teacher to tokenize dense depth into compact sequences, and (ii) integrate SAM2 based semantic segmentation tokens and VQ-VAE depth tokens inside the LLM so the model first emits spatial tokens and then answers. To stabilize depth token generation, we introduce novel composite depth-token objectives (marker, token, and count losses) and a soft-merging technique for differentiable reconstruction. We adopt a multi-task co-training strategy across diverse datasets, letting the model learn perception tokens to tackle multiple downstream tasks. Building on InternVL, Perceptio achieves state-of-the-art performance across benchmarks: improving referring expression segmentation by +0.8/+1.4/+1.1 cIoU on RefCOCO/+/g HardBLINK spatial understanding accuracy by 10.3%, and MMBench accuracy by 1.0%, demonstrating that explicit spatial chain-of-thought materially strengthens spatial grounding in LVLMs.
Paper Structure (28 sections, 13 equations, 7 figures, 5 tables, 1 algorithm)

This paper contains 28 sections, 13 equations, 7 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Vision-Language Models (LVLMs)
  4. Perception Guidance in LVLMs
  5. Methods
  6. Model Architecture
  7. Model Learning
  8. Perception Enhanced Generation.
  9. Depth Codebook.
  10. Model Inference
  11. Loss Functions
  12. LLM Loss
  13. 2D Supervision
  14. 3D Supervision
  15. Depth Token Generation Loss.
  16. ...and 13 more sections

Figures (7)

  • Figure 1: Comparison of Perceptio pipeline vs standard VLMs.
  • Figure 2: The Perceptio model. The language model takes text queries and image features as input to generate the desired text sequence (B). During training time, segmentation (A) and depth (C) teacher models supervise (via loss functions) the LVLM to accurately generate the intermediate perception tokens and the answer text output tokens.
  • Figure 3: Comparison between ground-truth and predicted outputs for a scene depicting two players engaged in an ultimate frisbee game. The first row shows the original image, ground-truth segmentation, and depth masks. The second row displays the question and the corresponding model predictions for text, segmentation mask, and depth map, demonstrating accurate recognition of the main objects and spatial depth relationships.
  • Figure 4: Perceptio evaluated on RefCOCO qualitative results. We compare our model with Sa2VA on a referring expression. Instance masks are colorized for visibility; depth maps are shown in grayscale (lighter = nearer). Our predictions align with semantic boundaries better than Sa2VA by capturing the expected depth layering across the image. Note: "×" denotes that no depth perception in Sa2VA.
  • Figure 5: Perceptio (Ours) predictions on HardBLINK dataset with overlay of color depth maps. Correct predictions in samples 1,2,3 and incorrect prediction in sample 4.
  • ...and 2 more figures