Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Models

Yuchen Hou, Lin Zhao

TL;DR

The LangGap benchmark is constructed, based on a four-dimensional semantic perturbation method -- varying instruction semantics while keeping the tabletop layout fixed -- revealing language understanding deficits in {\pi}0.5.

Abstract

Vision-Language-Action (VLA) models achieve over 95% success on standard benchmarks. However, through systematic experiments, we find that current state-of-the-art VLA models largely ignore language instructions. Prior work lacks: (1) systematic semantic perturbation diagnostics, (2) a benchmark that forces language understanding by design, and (3) linguistically diverse training data. This paper constructs the LangGap benchmark, based on a four-dimensional semantic perturbation method -- varying instruction semantics while keeping the tabletop layout fixed -- revealing language understanding deficits in π0.5. Existing benchmarks like LIBERO assign only one task per layout, underutilizing available objects and target locations; LangGap fully diversifies pick-and-place tasks under identical layouts, forcing models to truly understand language. Experiments show that targeted data augmentation can partially close the language gap -- success rate improves from 0% to 90% with single-task training, and 0% to 28% with multi-task training. However, as semantic diversity of extended tasks increases, model learning capacity proves severely insufficient; even trained tasks perform poorly. This reveals a fundamental challenge for VLA models in understanding diverse language instructions -- precisely the long-term value of LangGap.

LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Models

TL;DR

The LangGap benchmark is constructed, based on a four-dimensional semantic perturbation method -- varying instruction semantics while keeping the tabletop layout fixed -- revealing language understanding deficits in {\pi}0.5.

Abstract

Vision-Language-Action (VLA) models achieve over 95% success on standard benchmarks. However, through systematic experiments, we find that current state-of-the-art VLA models largely ignore language instructions. Prior work lacks: (1) systematic semantic perturbation diagnostics, (2) a benchmark that forces language understanding by design, and (3) linguistically diverse training data. This paper constructs the LangGap benchmark, based on a four-dimensional semantic perturbation method -- varying instruction semantics while keeping the tabletop layout fixed -- revealing language understanding deficits in π0.5. Existing benchmarks like LIBERO assign only one task per layout, underutilizing available objects and target locations; LangGap fully diversifies pick-and-place tasks under identical layouts, forcing models to truly understand language. Experiments show that targeted data augmentation can partially close the language gap -- success rate improves from 0% to 90% with single-task training, and 0% to 28% with multi-task training. However, as semantic diversity of extended tasks increases, model learning capacity proves severely insufficient; even trained tasks perform poorly. This reveals a fundamental challenge for VLA models in understanding diverse language instructions -- precisely the long-term value of LangGap.
Paper Structure (18 sections, 3 figures, 12 tables)

This paper contains 18 sections, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language-Action Models
  4. Manipulation Benchmarks and Robustness Evaluation
  5. Modality Imbalance and Architectural Solutions
  6. Distinction from Prior Work
  7. Method
  8. Semantic Perturbation Framework
  9. Diagnostic Evaluation
  10. LangGap Benchmark
  11. Training Data Collection
  12. Experiments
  13. Setup
  14. Results
  15. Analysis
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Four semantic perturbation types with example changes and observed success rates.
  • Figure 2: VLA performance under semantic perturbations. Original tasks achieve over 95% success on several suites. Semantic changes cause severe failures: Change Object 29.3%, Spatial Description 11.0%, Drawer Action 31.7%, and Change Target 0% across all 13 tasks.
  • Figure 3: Evaluation scenes from LIBERO suites that support multiple semantic variations.