Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking

David Emukpere, Romain Deffayet, Jean-Michel Renders

TL;DR

It is found that for representative VLA policies, including SmolVLA and $\pi_{0.5}$, execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success in harder regimes, suggesting that manipulation skill acquisition is decoupled from instruction following.

Abstract

Vision-language action (VLA) policies often report strong manipulation benchmark performance with relatively few demonstrations, but it remains unclear whether this reflects robust language-to-object grounding or reliance on object--location correlations that do not transfer beyond the training distribution. We present a controlled multi-object picking study that progressively increases object placement variability up to full workspace randomization and evaluates held-out object--location pairings that break familiar associations without increasing spatial difficulty. Across these stress tests and data scaling, we find that for representative VLA policies, including SmolVLA and $π_{0.5}$, execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success in harder regimes, suggesting that manipulation skill acquisition is decoupled from instruction following. We recommend augmenting manipulation benchmarks with task ladders and decomposed metrics that separately measure primitive execution and instruction-conditioned success to better diagnose instruction-grounded generalization.

Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking

TL;DR

It is found that for representative VLA policies, including SmolVLA and , execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success in harder regimes, suggesting that manipulation skill acquisition is decoupled from instruction following.

Abstract

Vision-language action (VLA) policies often report strong manipulation benchmark performance with relatively few demonstrations, but it remains unclear whether this reflects robust language-to-object grounding or reliance on object--location correlations that do not transfer beyond the training distribution. We present a controlled multi-object picking study that progressively increases object placement variability up to full workspace randomization and evaluates held-out object--location pairings that break familiar associations without increasing spatial difficulty. Across these stress tests and data scaling, we find that for representative VLA policies, including SmolVLA and , execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success in harder regimes, suggesting that manipulation skill acquisition is decoupled from instruction following. We recommend augmenting manipulation benchmarks with task ladders and decomposed metrics that separately measure primitive execution and instruction-conditioned success to better diagnose instruction-grounded generalization.
Paper Structure (29 sections, 4 figures, 6 tables)

This paper contains 29 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Environment Randomization in VLA Evaluation
  3. Related Work
  4. Vision-Language Action Policies.
  5. Manipulation Benchmarks, Generalization, and Shortcut Learning.
  6. Method
  7. Task Setup
  8. Difficulty Ladder
  9. Simulation Environment
  10. Data, training and evaluation
  11. Demonstration Data
  12. VLA policies
  13. Evaluation Metrics
  14. Experiments and Results
  15. Evaluation protocol.
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Object placement distributions under increasing randomization. Object placements progress from fixed regions with limited jitter to full workspace randomization. Each panel shows sampled initial object positions, with colors indicating different objects. In the jittered regimes, object placements remain largely separated across regions, enabling shortcut learning based on location. In contrast, full workspace randomization produces substantial overlap between object placement distributions, eliminating reliable positional cues.
  • Figure 2: Multi-object YCB environment in ManiSkill. The VLA observes two RGB camera streams: a wrist-mounted camera (top left) and a fixed frontal camera (top right). The bottom image shows the full environment for visualization only and is not provided as input to the policy.
  • Figure 3: Example of a trajectory produced by a $\pi_{0.5}$ model trained on $100k$ expert demonstrations in the Full Random environment.
  • Figure 4: Example of a trajectory produced by a $\pi_{0.5}$ model trained on $100k$ expert demonstrations in the Full Random environment.