Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models

Borong Zhang, Jiahao Li, Jiachen Shen, Yishuai Cai, Yuhao Zhang, Yuanpei Chen, Juntao Dai, Jiaming Ji, Yaodong Yang

TL;DR

VLA-Arena provides an open-source framework to benchmark Vision-Language-Action models with a structured, three-axis design that quantifies task difficulty across Task Structure, Language Command, and Visual Observation. It introduces CBDDL to formalize dynamic objects and safety constraints, and two orthogonal perturbation axes (WordNet-based language substitutions and cumulative visual disturbances) to diagnose robustness. The benchmark spans 170 tasks across 11 suites organized into Safety, Distractor, Extrapolation, and Long Horizon, and includes an end-to-end toolchain plus VLA-Arena-S/M/L datasets for standardized fine-tuning. Across autoregressive and continuous-action VLAs, the study reveals memorization over generalization, asymmetric robustness to perturbations, a pervasive safety-performance trade-off, and a lack of long-horizon skill composition, highlighting critical gaps for real-world deployment and providing a foundation for more robust, safe, and generalizable robotic agents.

Abstract

While Vision-Language-Action models (VLAs) are rapidly advancing towards generalist robot policies, it remains difficult to quantitatively understand their limits and failure modes. To address this, we introduce a comprehensive benchmark called VLA-Arena. We propose a novel structured task design framework to quantify difficulty across three orthogonal axes: (1) Task Structure, (2) Language Command, and (3) Visual Observation. This allows us to systematically design tasks with fine-grained difficulty levels, enabling a precise measurement of model capability frontiers. For Task Structure, VLA-Arena's 170 tasks are grouped into four dimensions: Safety, Distractor, Extrapolation, and Long Horizon. Each task is designed with three difficulty levels (L0-L2), with fine-tuning performed exclusively on L0 to assess general capability. Orthogonal to this, language (W0-W4) and visual (V0-V4) perturbations can be applied to any task to enable a decoupled analysis of robustness. Our extensive evaluation of state-of-the-art VLAs reveals several critical limitations, including a strong tendency toward memorization over generalization, asymmetric robustness, a lack of consideration for safety constraints, and an inability to compose learned skills for long-horizon tasks. To foster research addressing these challenges and ensure reproducibility, we provide the complete VLA-Arena framework, including an end-to-end toolchain from task definition to automated evaluation and the VLA-Arena-S/M/L datasets for fine-tuning. Our benchmark, data, models, and leaderboard are available at https://vla-arena.github.io.

VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models

TL;DR

VLA-Arena provides an open-source framework to benchmark Vision-Language-Action models with a structured, three-axis design that quantifies task difficulty across Task Structure, Language Command, and Visual Observation. It introduces CBDDL to formalize dynamic objects and safety constraints, and two orthogonal perturbation axes (WordNet-based language substitutions and cumulative visual disturbances) to diagnose robustness. The benchmark spans 170 tasks across 11 suites organized into Safety, Distractor, Extrapolation, and Long Horizon, and includes an end-to-end toolchain plus VLA-Arena-S/M/L datasets for standardized fine-tuning. Across autoregressive and continuous-action VLAs, the study reveals memorization over generalization, asymmetric robustness to perturbations, a pervasive safety-performance trade-off, and a lack of long-horizon skill composition, highlighting critical gaps for real-world deployment and providing a foundation for more robust, safe, and generalizable robotic agents.

Abstract

While Vision-Language-Action models (VLAs) are rapidly advancing towards generalist robot policies, it remains difficult to quantitatively understand their limits and failure modes. To address this, we introduce a comprehensive benchmark called VLA-Arena. We propose a novel structured task design framework to quantify difficulty across three orthogonal axes: (1) Task Structure, (2) Language Command, and (3) Visual Observation. This allows us to systematically design tasks with fine-grained difficulty levels, enabling a precise measurement of model capability frontiers. For Task Structure, VLA-Arena's 170 tasks are grouped into four dimensions: Safety, Distractor, Extrapolation, and Long Horizon. Each task is designed with three difficulty levels (L0-L2), with fine-tuning performed exclusively on L0 to assess general capability. Orthogonal to this, language (W0-W4) and visual (V0-V4) perturbations can be applied to any task to enable a decoupled analysis of robustness. Our extensive evaluation of state-of-the-art VLAs reveals several critical limitations, including a strong tendency toward memorization over generalization, asymmetric robustness, a lack of consideration for safety constraints, and an inability to compose learned skills for long-horizon tasks. To foster research addressing these challenges and ensure reproducibility, we provide the complete VLA-Arena framework, including an end-to-end toolchain from task definition to automated evaluation and the VLA-Arena-S/M/L datasets for fine-tuning. Our benchmark, data, models, and leaderboard are available at https://vla-arena.github.io.
Paper Structure (84 sections, 1 equation, 3 figures, 21 tables)

This paper contains 84 sections, 1 equation, 3 figures, 21 tables.

Table of Contents

  1. Introduction
  2. Structured Task Design
  3. Task Structure: Beyond Memorization
  4. Constrained BDDL.
  5. Language Command: Semantic Grounding
  6. Principled Word Substitution.
  7. Visual Observation: Perceptual Change
  8. A Cumulative Hierarchy of Visual Difficulty.
  9. Task Suites in VLA-Arena
  10. Safety.
  11. Distractor.
  12. Extrapolation.
  13. Long Horizon.
  14. Experiments
  15. Experimental Setup
  16. ...and 69 more sections

Figures (3)

  • Figure 1: Performance Degradation of VLA Models under Language and Visual Perturbations. Robustness is evaluated along two orthogonal axes: language perturbations (W0–W4) with increasingly strong semantic substitutions and visual perturbations (V0–V4) with cumulative perceptual distortions. Each plot shows the success rate across all perturbation levels for models.
  • Figure 2: Impact of Language Instruction on Model Performance Across VLA-Arena and LIBERO Benchmarks.
  • Figure 3: Visualization of Task Distributions across VLA-Arena and LIBERO.