Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QDepth-VLA: Quantized Depth Prediction as Auxiliary Supervision for Vision-Language-Action Models

Yixuan Li, Yuhui Chen, Mingcai Zhou, Haoran Li, Zhengtao Zhang, Dongbin Zhao

TL;DR

<3-5 sentence high-level summary>QDepth-VLA tackles the gap in geometric reasoning for Vision-Language-Action models by introducing quantized depth supervision through a dedicated depth expert. The method relies on a VQ-VAE-based depth codebook and a hybrid attention mechanism to incorporate discrete depth tokens without disturbing the pretrained vision-language backbone, enabling geometry-aware policy learning. Across LIBERO, Simpler benchmarks, and real-world Piper-arm experiments, QDepth-VLA yields consistent improvements in spatial reasoning and long-horizon manipulation, approaching multi-view performance even with single-view inputs. The work demonstrates the practicality of latent, discretized depth representations for robust, geometry-grounded VLA control and points to future work on predicting future depth tokens and more efficient depth representations.

Abstract

Spatial perception and reasoning are crucial for Vision-Language-Action (VLA) models to accomplish fine-grained manipulation tasks. However, existing approaches often lack the ability to understand and reason over the essential 3D structures necessary for precise control. To address this limitation, we propose QDepth-VLA, a general framework that augments VLA models with an auxiliary depth prediction task. A dedicated depth expert is designed to predict quantized latent tokens of depth maps obtained from a VQ-VAE encoder, enabling the model to learn depth-aware representations that capture critical geometric cues. Experimental results on the simulation benchmarks and real-world tasks demonstrate that QDepth-VLA yields strong spatial reasoning and competitive performance on manipulation tasks.

QDepth-VLA: Quantized Depth Prediction as Auxiliary Supervision for Vision-Language-Action Models

TL;DR

<3-5 sentence high-level summary>QDepth-VLA tackles the gap in geometric reasoning for Vision-Language-Action models by introducing quantized depth supervision through a dedicated depth expert. The method relies on a VQ-VAE-based depth codebook and a hybrid attention mechanism to incorporate discrete depth tokens without disturbing the pretrained vision-language backbone, enabling geometry-aware policy learning. Across LIBERO, Simpler benchmarks, and real-world Piper-arm experiments, QDepth-VLA yields consistent improvements in spatial reasoning and long-horizon manipulation, approaching multi-view performance even with single-view inputs. The work demonstrates the practicality of latent, discretized depth representations for robust, geometry-grounded VLA control and points to future work on predicting future depth tokens and more efficient depth representations.

Abstract

Spatial perception and reasoning are crucial for Vision-Language-Action (VLA) models to accomplish fine-grained manipulation tasks. However, existing approaches often lack the ability to understand and reason over the essential 3D structures necessary for precise control. To address this limitation, we propose QDepth-VLA, a general framework that augments VLA models with an auxiliary depth prediction task. A dedicated depth expert is designed to predict quantized latent tokens of depth maps obtained from a VQ-VAE encoder, enabling the model to learn depth-aware representations that capture critical geometric cues. Experimental results on the simulation benchmarks and real-world tasks demonstrate that QDepth-VLA yields strong spatial reasoning and competitive performance on manipulation tasks.
Paper Structure (31 sections, 8 equations, 3 figures, 6 tables)

This paper contains 31 sections, 8 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. RELATED WORKS
  3. 3D-Enhanced VLA
  4. Auxiliary Visual Reasoning Tasks for VLA
  5. Methodology
  6. Depth Annotation
  7. VQ-VAE Reconstruction
  8. QDepth-VLA Architecture
  9. Co-Training Procedures
  10. Quantized Depth Supervision
  11. Action Modeling
  12. Co-Training Objectives
  13. Optimization Setup
  14. EXPERIMENTS
  15. Simulation Experiments
  16. ...and 16 more sections

Figures (3)

  • Figure 1: An overview of QDepth-VLA. (a) The overall architecture and training pipeline, where depth supervision is incorporated via a depth expert and latent prediction module. In co-training, the VQ-VAE Oord2017NeuralDR encoder and codebook are frozen, while PaLI-Gemma 3B Beyer2024PaliGemmaAV, the action expert, depth expert, SigLIP Zhai2023SigmoidLF, and tokenizer are trainable. (b) The proposed hybrid attention mask, which integrates depth and visual tokens to enhance spatial reasoning and manipulation performance.
  • Figure 2: An overview of the main camera view in our real-world task. The environment presents significant challenges for policy learning, including complex lighting conditions, low visual contrast between the gripper and tabletop, and various environmental disturbances that can obscure critical geometric details.
  • Figure 3: Depth reconstruction results from the QDepth-VLA. Features predicted by depth expert are decoded using a trained VQ-VAE decoder. The reconstructions demonstrate QDepth-VLA's ability to learn critical depth map features, including object and gripper boundaries, underscoring the success of its depth supervision.