A2Eval: Agentic and Automated Evaluation for Embodied Brain

TL;DR

The paper tackles the prohibitive cost and bias in evaluating embodied vision-language models by replacing static expert benchmarks with an agentic automatic evaluation framework. It introduces two collaborative agents: a Data Agent that autonomously induces a principled capability taxonomy and constructs a compact, diverse benchmark, and an Eval Agent that synthesizes executable inference and scoring pipelines for fully autonomous evaluation. Across 10 benchmarks and 13 models, A2Eval compresses benchmarks by $85\%$, reduces overall costs by $77\%$, and delivers a $4.6\times$ speedup while preserving evaluation quality, with human-alignment and ranking fidelity improving to $ ho=0.85$ and $ au=0.81$, respectively. The results demonstrate high-fidelity, low-cost embodied assessment that facilitates rapid iterative development, and the authors plan to release code and data to catalyze further adoption.

Abstract

Current embodied VLM evaluation relies on static, expert-defined, manually annotated benchmarks that exhibit severe redundancy and coverage imbalance. This labor intensive paradigm drains computational and annotation resources, inflates costs, and distorts model rankings, ultimately stifling iterative development. To address this, we propose Agentic Automatic Evaluation (A2Eval), the first agentic framework that automates benchmark curation and evaluation through two collaborative agents. The Data Agent autonomously induces capability dimensions and assembles a balanced, compact evaluation suite, while the Eval Agent synthesizes and validates executable evaluation pipelines, enabling fully autonomous, high-fidelity assessment. Evaluated across 10 benchmarks and 13 models, A2Eval compresses evaluation suites by 85%, reduces overall computational costs by 77%, and delivers a 4.6x speedup while preserving evaluation quality. Crucially, A2Eval corrects systematic ranking biases, improves human alignment to Spearman's rho=0.85, and maintains high ranking fidelity (Kendall's tau=0.81), establishing a new standard for high-fidelity, low-cost embodied assessment. Our code and data will be public soon.

Figures (6)

• Figure 1: (a): Capacity distributions across existing expert-defined and manually annotated embodied VLM benchmarks reveal redundant yet sparse coverage, leading to ranking distortion and excessive evaluation costs. (b): A2Eval replaces this expert-defined, annotation-heavy paradigm with an automated suite that maintains capability coverage while compressing benchmarks, achieving 4.6$\times$ speedup and improved human alignment. (Details in Tables \ref{['tab:rank']} and \ref{['tab:model_efficiency']}.)
• Figure 2: Overview of A2Eval. The left half shows the Data Agent with three roles, Proposer, Reviewer, and Assigner, which constructs compact, balanced benchmarks. The right half shows the Eval Agent with two roles, Evaluator and Scorer, which produces model predictions and scores. Each role is represented by a distinct color.
• Figure 4: Heatmap of inter-benchmark similarity. The results reveal severe redundancy in the current evaluation ecosystem, with benchmarks exhibiting up to 91% overlap (e.g., RefSpatial vs. Where2Place).
• Figure 5: Capability distribution across dimensions before and after diversity-aware sampling. The retained set exhibits a substantially more balanced coverage compared to the highly skewed source distribution.
• Figure 6: UMAP visualization of sample embeddings across all dimensions.