Trustworthy Evaluation of Robotic Manipulation: A New Benchmark and AutoEval Methods

TL;DR

This work addresses the credibility gap in robotic manipulation evaluation by distinguishing execution quality from source authenticity. It introduces Eval-Actions, a dataset that includes failure cases and mixed policy/teleoperation trajectories with fine-grained action quality signals annotated via Expert Grading, Rank-Guided preferences, and Chain-of-Thought. Complementing this, AutoEval provides two architectures: AutoEval-S leverages Spatio-Temporal Aggregation for precise scoring and a physics-aware prompt, while AutoEval-P employs Group Relative Policy Optimization to align long-horizon reasoning with scores. On the Eval-Actions benchmark, AutoEval achieves up to SRCC $=0.84$ and 99.6% source discrimination accuracy, establishing a robust standard for trustworthy embodied evaluation with practical implications for benchmarking and model assessment in VA/VLA robotic manipulation.

Abstract

Driven by the rapid evolution of Vision-Action and Vision-Language-Action models, imitation learning has significantly advanced robotic manipulation capabilities. However, evaluation methodologies have lagged behind, hindering the establishment of Trustworthy Evaluation for these behaviors. Current paradigms rely on binary success rates, failing to address the critical dimensions of trust: Source Authenticity (i.e., distinguishing genuine policy behaviors from human teleoperation) and Execution Quality (e.g., smoothness and safety). To bridge these gaps, we propose a solution that combines the Eval-Actions benchmark and the AutoEval architecture. First, we construct the Eval-Actions benchmark to support trustworthiness analysis. Distinct from existing datasets restricted to successful human demonstrations, Eval-Actions integrates VA and VLA policy execution trajectories alongside human teleoperation data, explicitly including failure scenarios. This dataset is structured around three core supervision signals: Expert Grading (EG), Rank-Guided preferences (RG), and Chain-of-Thought (CoT). Building on this, we propose the AutoEval architecture: AutoEval leverages Spatio-Temporal Aggregation for semantic assessment, augmented by an auxiliary Kinematic Calibration Signal to refine motion smoothness; AutoEval Plus (AutoEval-P) incorporates the Group Relative Policy Optimization (GRPO) paradigm to enhance logical reasoning capabilities. Experiments show AutoEval achieves Spearman's Rank Correlation Coefficients (SRCC) of 0.81 and 0.84 under the EG and RG protocols, respectively. Crucially, the framework possesses robust source discrimination capabilities, distinguishing between policy-generated and teleoperated videos with 99.6% accuracy, thereby establishing a rigorous standard for trustworthy robotic evaluation. Our project and code are available at https://term-bench.github.io/.

Figures (7)

• Figure 1: The Crisis of Evaluation Credibility and the Proposed Trustworthy Evaluation Solution.(Top) The Crisis: We identify two critical sources of ambiguity obstructing trustworthy evaluation: Gap 1 (Ambiguity in Execution Quality), where binary metrics mask shaky or unsafe execution (visualized as "Jerky Success" vs. "Smooth Success"), and Gap 2 (Ambiguity in Source Authenticity), where the provenance of "successful" demonstrations is unverifiable. (Bottom) Proposed Solution: Our Trustworthy Evaluation Framework bridges these gaps. Powered by the Eval-Actions Benchmark and the AutoEval Architecture (depicted as the green robot optimized via Supervised Fine-Tuning (SFT)), the system achieves precise Fine-Grained Action Quality assessment (SRCC 0.84) and robust Source Authenticity verification (99.6%), as shown in the green box. This significantly outperforms standard Vision-Language Models (VLMs) without SFT (red box) to ensure evaluation credibility.
• Figure 2: Overview of the Eval-Actions Benchmark. The figure visualizes the dataset structure: (Left) Task Diversity: Representative snapshots from our 150+ scenarios, covering both single-arm interactions (e.g., “Throw away trash") and complex bimanual coordination (e.g., “Tidy medicine box"). (Middle) Detailed Case Study: A specific instantiation of the “Throw away trash" task shown on the left. Crucially, each task encompasses diverse demonstration data ranging from high-quality successes to failure scenarios—exemplified here by contrasting smooth teleoperation with jerky policy behaviors. (Right) Data Composition: The stack enumerates the dense multimodal signals encapsulated within each episode. This includes raw sensory data (RGB, Depth), precise kinematic records (7/14-DoF Joint Trajectories), and the Fine-Grained Quality Radar Chart, which explicitly quantifies the four core dimensions (Success, Smoothness, Safety, Efficiency) to enable diagnostic assessment.
• Figure 3: Representative Task Statistics of the Eval-Actions. The top chart illustrates the distribution of the number of demonstrations for representative tasks, while the bottom chart displays the total duration (in seconds) for each corresponding task. These tasks cover diverse manipulation scenarios, including both single-arm and dual-arm operations.
• Figure 4: Distribution of Expert Grading across the Eval-Actions Small subset.
• Figure 5: Overview of the proposed AutoEval framework. The system processes a robot manipulation video sequence (e.g., 32 frames) alongside kinematic prompts. Top (AutoEval-S): Designed for Expert Grading and Rank-Guided tasks, this branch employs a Spatio-Temporal Aggregation Strategy to compress high-frequency motion details into composite visual tokens. It generates structured text predictions; following format decomposition, the model is optimized via Supervised Fine-Tuning (SFT) using Cross-Entropy Loss. Bottom (AutoEval-P): Tailored for Chain-of-Thought (CoT) reasoning, this branch adopts the Group Relative Policy Optimization (GRPO)GPROguo2025deepseek paradigm. The policy model generates multiple reasoning paths (containing <think> tokens), optimized against a hybrid reward function comprising content accuracy ($r_{Content}$) and format constraints ($r_{Format}$) to enhance physical reasoning capabilities.