Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GLaD: Geometric Latent Distillation for Vision-Language-Action Models

Minghao Guo, Meng Cao, Jiachen Tao, Rongtao Xu, Yan Yan, Xiaodan Liang, Ivan Laptev, Xiaojun Chang

TL;DR

The paper tackles the lack of geometric reasoning in Vision-Language-Action models by introducing GLaD, a geometry-aware pretraining framework that distills 3D priors from a frozen VGGT into the LLM-based VLA backbone. GLaD trains with a joint objective that aligns LLM image-token representations with VGGT geometry features, then fine-tunes task-specific components via LoRA. On LIBERO, GLaD achieves 94.1% average success, surpassing UniVLA, and demonstrates enhanced robustness to appearance perturbations in LIBERO-PRO, especially for object-centric tasks. Ablation studies confirm the critical roles of VGGT as geometry encoder, late-stage alignment, and a geometry-informed fusion strategy. Collectively, the results show geometry-aware pretraining can improve spatial reasoning and policy generalization without depth sensors or explicit 3D annotations, advancing robust manipulation in vision-language systems.

Abstract

Most existing Vision-Language-Action (VLA) models rely primarily on RGB information, while ignoring geometric cues crucial for spatial reasoning and manipulation. In this work, we introduce GLaD, a geometry-aware VLA framework that incorporates 3D geometric priors during pretraining through knowledge distillation. Rather than distilling geometric features solely into the vision encoder, we align the LLM's hidden states corresponding to visual tokens with features from a frozen geometry-aware vision transformer (VGGT), ensuring that geometric understanding is deeply integrated into the multimodal representations that drive action prediction. Pretrained on the Bridge dataset with this geometry distillation mechanism, GLaD achieves 94.1% average success rate across four LIBERO task suites, outperforming UniVLA (92.5%) which uses identical pretraining data. These results validate that geometry-aware pretraining enhances spatial reasoning and policy generalization without requiring explicit depth sensors or 3D annotations.

GLaD: Geometric Latent Distillation for Vision-Language-Action Models

TL;DR

The paper tackles the lack of geometric reasoning in Vision-Language-Action models by introducing GLaD, a geometry-aware pretraining framework that distills 3D priors from a frozen VGGT into the LLM-based VLA backbone. GLaD trains with a joint objective that aligns LLM image-token representations with VGGT geometry features, then fine-tunes task-specific components via LoRA. On LIBERO, GLaD achieves 94.1% average success, surpassing UniVLA, and demonstrates enhanced robustness to appearance perturbations in LIBERO-PRO, especially for object-centric tasks. Ablation studies confirm the critical roles of VGGT as geometry encoder, late-stage alignment, and a geometry-informed fusion strategy. Collectively, the results show geometry-aware pretraining can improve spatial reasoning and policy generalization without depth sensors or explicit 3D annotations, advancing robust manipulation in vision-language systems.

Abstract

Most existing Vision-Language-Action (VLA) models rely primarily on RGB information, while ignoring geometric cues crucial for spatial reasoning and manipulation. In this work, we introduce GLaD, a geometry-aware VLA framework that incorporates 3D geometric priors during pretraining through knowledge distillation. Rather than distilling geometric features solely into the vision encoder, we align the LLM's hidden states corresponding to visual tokens with features from a frozen geometry-aware vision transformer (VGGT), ensuring that geometric understanding is deeply integrated into the multimodal representations that drive action prediction. Pretrained on the Bridge dataset with this geometry distillation mechanism, GLaD achieves 94.1% average success rate across four LIBERO task suites, outperforming UniVLA (92.5%) which uses identical pretraining data. These results validate that geometry-aware pretraining enhances spatial reasoning and policy generalization without requiring explicit depth sensors or 3D annotations.

Paper Structure

This paper contains 26 sections, 4 equations, 13 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Geometry Distillation
  5. VGGT Feature Extractor
  6. Feature Alignment Network
  7. Training Strategy
  8. Stage 1: Pretraining with Geometry Distillation
  9. Stage 2: Posttraining on Downstream Tasks
  10. Experiments
  11. Datasets and Benchmarks
  12. Bridge Dataset
  13. LIBERO Benchmark and LIBERO Dataset
  14. LIBERO-PRO Benchmark
  15. Training Protocol
  16. ...and 11 more sections

Figures (13)

  • Figure 3: Robustness comparison across LIBERO suites under five perturbation types. We compare GLaD against UniVLA on four LIBERO suites: GOAL, SPATIAL, Long-horizon (10), and OBJECT. Ori: Original tasks; Obj: Object perturbations (color, texture, size); Pos: Position perturbations; Sem: Semantic perturbations (language); Task: Task perturbations. Success rates (%) averaged over 50 episodes per task. GLaD demonstrates significant improvements in object perturbation robustness, particularly on GOAL (81% vs 62%) and Long (54% vs 47%).
  • Figure : VLA w/o geometry
  • Figure : (a)
  • Figure : UniVLA
  • Figure : VLA w/o geometry
  • ...and 8 more figures