Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies

Adina Yakefu, Bin Xie, Chongyang Xu, Enwen Zhang, Erjin Zhou, Fan Jia, Haitao Yang, Haoqiang Fan, Haowei Zhang, Hongyang Peng, Jing Tan, Junwen Huang, Kai Liu, Kaixin Liu, Kefan Gu, Qinglun Zhang, Ruitao Zhang, Saike Huang, Shen Cheng, Shuaicheng Liu, Tiancai Wang, Tiezhen Wang, Wei Sun, Wenbin Tang, Yajun Wei, Yang Chen, Youqiang Gui, Yucheng Zhao, Yunchao Ma, Yunfei Wei, Yunhuan Yang, Yutong Guo, Ze Chen, Zhengyuan Du, Ziheng Zhang, Ziming Liu, Ziwei Yan

TL;DR

RoboChallenge provides a real-robot, online evaluation framework to benchmark vision-language-action policies at scale on a fleet of 10 heterogeneous robots. It introduces a remote-robot paradigm with timestamped observations, asynchronous action queues, and accessible demonstration data to enable fair, reproducible, large-scale testing beyond simulators. The Table30 benchmark comprises 30 diverse tasks around a table, evaluated with a two-tier training setting (Task-specific and Generalist) and a progress-based grading scheme across 10 rollouts per task. Empirical results show strong performance by advanced baselines, particularly Pi0.5, while identifying persistent challenges in temporal reasoning and deformable object manipulation that motivate future improvements in realism-focused benchmarking and protocol design.

Abstract

Testing on real machines is indispensable for robotic control algorithms. In the context of learning-based algorithms, especially VLA models, demand for large-scale evaluation, i.e. testing a large number of models on a large number of tasks, is becoming increasingly urgent. However, doing this right is highly non-trivial, especially when scalability and reproducibility is taken into account. In this report, we describe our methodology for constructing RoboChallenge, an online evaluation system to test robotic control algorithms, and our survey of recent state-of-the-art VLA models using our initial benchmark Table30.

RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies

TL;DR

RoboChallenge provides a real-robot, online evaluation framework to benchmark vision-language-action policies at scale on a fleet of 10 heterogeneous robots. It introduces a remote-robot paradigm with timestamped observations, asynchronous action queues, and accessible demonstration data to enable fair, reproducible, large-scale testing beyond simulators. The Table30 benchmark comprises 30 diverse tasks around a table, evaluated with a two-tier training setting (Task-specific and Generalist) and a progress-based grading scheme across 10 rollouts per task. Empirical results show strong performance by advanced baselines, particularly Pi0.5, while identifying persistent challenges in temporal reasoning and deformable object manipulation that motivate future improvements in realism-focused benchmarking and protocol design.

Abstract

Testing on real machines is indispensable for robotic control algorithms. In the context of learning-based algorithms, especially VLA models, demand for large-scale evaluation, i.e. testing a large number of models on a large number of tasks, is becoming increasingly urgent. However, doing this right is highly non-trivial, especially when scalability and reproducibility is taken into account. In this report, we describe our methodology for constructing RoboChallenge, an online evaluation system to test robotic control algorithms, and our survey of recent state-of-the-art VLA models using our initial benchmark Table30.
Paper Structure (21 sections, 10 figures, 2 tables)

This paper contains 21 sections, 10 figures, 2 tables.

Table of Contents

  1. Introduction
  2. RoboChallenge
  3. Online Interface for Serving Robots
  4. The Robot Platforms
  5. Evaluation Protocol
  6. Variation of Testers
  7. Visual Task Reproduction
  8. Background and Environmental Issues
  9. Stability v.s. Fairness
  10. Limitation and Known Issues
  11. Table30 Benchmark
  12. The Tasks
  13. Grading Protocol
  14. Designing Tasks that Differentiates
  15. Results on Table30
  16. ...and 6 more sections

Figures (10)

  • Figure 1: We served our robots online. A set of low-level api is formalized to provide the exact timestamp of observations and state of the action queue to enable fine-grained control. No docker images or model checkpoints are needed to be exchanged.
  • Figure 2: Left: Thumbnails of the tasks. Right: ranklist of the baseline methods. Our first benchmark is a 30-task static armed robot testing set. It challenges a variety of aspects of the learning algorithms. We measured the end-to-end task-level success rate and a score that measures the partial progress of the tasks, and see a clear distinction between the models. Models marked by rc_baseline are finetuned by the authors of the report. Other models are finetuned by a group of college volunteers.
  • Figure 3: Variation of success rates caused by testers. We picked two tasks and tried three different testers, each with (1) experienced tester: the same one that collected the training data (2) ignorant tester: totally fresh tester seeing the prompt and props for the first time (3) adaptive tester: a tester with algorithm experience and managing to "improve" the success rate as much as he/she could.
  • Figure 4: The "Sweet-spot Effect". We plot the positions of the box chosen by an "adaptive tester", and use green and red color to indicate a successful or failed task. The tester managed to find the location and orientation of the box that the task is more likely to succeed, and exploited this for maximal performance. This biased the test.
  • Figure 5: The tester's user interface for Visual Task Reproduction. A reference image is superimposed on the live camera stream. The tester is instructed to adjust the position of the objects and other factors so that the images match.
  • ...and 5 more figures