The Great March 100: 100 Detail-oriented Tasks for Evaluating Embodied AI Agents

TL;DR

The paper addresses biases in existing robotic task datasets that emphasize a narrow set of skills and inconsistent evaluations. It proposes GM-100, a 100-task, long-tail benchmark derived from human-object interaction primitives and object affordances, generated with LLMs and filtered by experts, and paired with a 13K-trajectory teleoperation dataset collected on two platforms. The contributions include a systematic task-design pipeline, an open-source dataset and task list, and baseline evaluations showing GM-100 tasks are feasible yet discriminative across methods, thereby enabling fair cross-study comparisons. The work aims to foster an open, community-driven benchmarking ecosystem (GM-X) that can guide future embodied AI research and standardize evaluation practices.

Abstract

Recently, with the rapid development of robot learning and imitation learning, numerous datasets and methods have emerged. However, these datasets and their task designs often lack systematic consideration and principles. This raises important questions: Do the current datasets and task designs truly advance the capabilities of robotic agents? Do evaluations on a few common tasks accurately reflect the differentiated performance of various methods proposed by different teams and evaluated on different tasks? To address these issues, we introduce the Great March 100 (\textbf{GM-100}) as the first step towards a robot learning Olympics. GM-100 consists of 100 carefully designed tasks that cover a wide range of interactions and long-tail behaviors, aiming to provide a diverse and challenging set of tasks to comprehensively evaluate the capabilities of robotic agents and promote diversity and complexity in robot dataset task designs. These tasks are developed through systematic analysis and expansion of existing task designs, combined with insights from human-object interaction primitives and object affordances. We collect a large amount of trajectory data on different robotic platforms and evaluate several baseline models. Experimental results demonstrate that the GM-100 tasks are 1) feasible to execute and 2) sufficiently challenging to effectively differentiate the performance of current VLA models. Our data and code are available at https://rhos.ai/research/gm-100.

Figures (6)

• Figure 1: Word cloud of task descriptions of existing works.
• Figure 2: Verb frequency distribution in previous task descriptions.
• Figure 4: The construction pipeline of the GM-100 benchmark. The process begins with collecting existing robot tasks, followed by a semantic expansion using HAKE li2019hakehumanactivityknowledge and LLM-based generation to cover long-tail interactions. The candidates then undergo a rigorous hybrid filtration by LLMs and human experts to ensure hardware feasibility and data collection friendliness. Finally, 100 high-priority tasks are selected and instantiated with detailed interaction criteria and template videos.
• Figure 5: GM-100 Dataset. Two distinct robotic platforms are utilized for data collection and evaluation. For Tasks 1–10, we collect 130 trajectories per task on both platforms, whereas for Tasks 11–100, data is collected exclusively on the Cobot Magic platform. To ensure the 100 training trajectories and 30 testing trajectories share a similar data distribution, we strive to maintain consistency in the environments and object configurations used for both data collection and evaluation. (Note: The included data distribution figure is schematic and intended solely for illustration purposes; it does not depict the exact statistical distribution.)
• Figure 6: Partial Success Rate on Cobot Magic Platform. The color intensity in the heatmap indicates the PSR. Task Details are provided at https://rhos.ai/research/gm-100/tasks due to the space limit. Results on more baselines will be gradually released over time due to the time-consuming nature of real-world robot testing. Detailed PSR can be found at https://rhos.ai/research/gm-100/psr.