Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models

Suhwan Choi, Yunsung Lee, Yubeen Park, Chris Dongjoo Kim, Ranjay Krishna, Dieter Fox, Youngjae Yu

Abstract

Vision Language Action VLA models are typically evaluated using per benchmark scripts maintained independently by each model repository, leading to duplicated code, dependency conflicts, and underspecified protocols. We present vla eval, an open source evaluation harness that decouples model inference from benchmark execution through a WebSocket msgpack protocol with Docker based environment isolation. Models integrate once by implementing a single predict() method; benchmarks integrate once via a four method interface; the full cross evaluation matrix works automatically. A complete evaluation requires only two commands: vla eval serve and vla eval run. The framework supports 13 simulation benchmarks and six model servers. Parallel evaluation via episode sharding and batch inference achieves a 47x throughput improvement, completing 2000 LIBERO episodes in about 18 minutes. Using this infrastructure, we conduct a reproducibility audit of a published VLA model across three benchmarks, finding that all three closely reproduce published values while uncovering undocumented requirements ambiguous termination semantics and hidden normalization statistics that can silently distort results. We additionally release a VLA leaderboard aggregating 657 published results across 17 benchmarks. Framework, evaluation configs, and all reproduction results are publicly available.

vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models

Abstract

Vision Language Action VLA models are typically evaluated using per benchmark scripts maintained independently by each model repository, leading to duplicated code, dependency conflicts, and underspecified protocols. We present vla eval, an open source evaluation harness that decouples model inference from benchmark execution through a WebSocket msgpack protocol with Docker based environment isolation. Models integrate once by implementing a single predict() method; benchmarks integrate once via a four method interface; the full cross evaluation matrix works automatically. A complete evaluation requires only two commands: vla eval serve and vla eval run. The framework supports 13 simulation benchmarks and six model servers. Parallel evaluation via episode sharding and batch inference achieves a 47x throughput improvement, completing 2000 LIBERO episodes in about 18 minutes. Using this infrastructure, we conduct a reproducibility audit of a published VLA model across three benchmarks, finding that all three closely reproduce published values while uncovering undocumented requirements ambiguous termination semantics and hidden normalization statistics that can silently distort results. We additionally release a VLA leaderboard aggregating 657 published results across 17 benchmarks. Framework, evaluation configs, and all reproduction results are publicly available.
Paper Structure (12 sections, 5 figures, 2 tables)

This paper contains 12 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Framework Design
  3. Architecture
  4. Supported Benchmarks and Models
  5. Parallel Evaluation
  6. Reproducibility Audit
  7. Results
  8. Sources of Discrepancy
  9. VLA Leaderboard
  10. Curation
  11. Cross-Benchmark Analysis
  12. Discussion

Figures (5)

  • Figure 1: System architecture. The model server runs on the host; each benchmark runs in an isolated Docker container with optional GPU access for rendering. A SyncEpisodeRunner orchestrates the observe$\to$act$\to$step loop via a Connection (WebSocket + msgpack client).
  • Figure 2: Demand/supply throughput for LIBERO + CogACT on H100. Dashed lines show supply ceilings $\mu(B)$ at each batch size. The operating point $N^*\!=\!50$ uses 78% of the supply capacity at $B\!=\!16$, leaving headroom to absorb burst arrivals and prevent queue buildup; beyond $N\!=\!80$, environment overhead causes throughput to drop.
  • Figure 3: Wall-clock evaluation time: sequential vs. batch parallel. LIBERO: 2,000 episodes, 50 shards, $B\!=\!16$. CALVIN: 1,000 sequences, 16 shards. SimplerEnv: 288 episodes (3 seeds), 16 shards.
  • Figure 4: VLA leaderboard (17 benchmarks, \urlLeaderboard). Shown: models with $>$10 citations. Filterable by benchmark and model; contributions via pull request.
  • Figure 5: Distribution of benchmark coverage per model. 81% of the 509+ models in the leaderboard are evaluated on only one benchmark. Only 3 models (0.6%) are evaluated on 5 or more. Counts are lower bounds, as models may evaluate on benchmarks not tracked in the leaderboard. A unified evaluation framework is essential to make systematic cross-benchmark comparison feasible.