Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RB2: Robotic Manipulation Benchmarking with a Twist

Sudeep Dasari, Jianren Wang, Joyce Hong, Shikhar Bahl, Yixin Lin, Austin Wang, Abitha Thankaraj, Karanbir Chahal, Berk Calli, Saurabh Gupta, David Held, Lerrel Pinto, Deepak Pathak, Vikash Kumar, Abhinav Gupta

TL;DR

RB2 rethinks robotic manipulation benchmarking by coupling a fixed task suite with a model zoo of baselines and a central repository to enable local rankings that generalize into global rankings across labs. It introduces four SHAP-inspired tasks, five baseline methods, and a concrete protocol for data collection, training, and evaluation, enabling statistically grounded comparisons. Across two lab spaces, simple open-loop imitation often rivals or outperforms more complex methods like NDPs and MOReL, highlighting the gap between strong benchmarks and real-world performance. The global ranking mechanism democratizes benchmarking, allowing the community to collectively converge on credible baselines and robust progress in robotic manipulation.

Abstract

Benchmarks offer a scientific way to compare algorithms using objective performance metrics. Good benchmarks have two features: (a) they should be widely useful for many research groups; (b) and they should produce reproducible findings. In robotic manipulation research, there is a trade-off between reproducibility and broad accessibility. If the benchmark is kept restrictive (fixed hardware, objects), the numbers are reproducible but the setup becomes less general. On the other hand, a benchmark could be a loose set of protocols (e.g. object sets) but the underlying variation in setups make the results non-reproducible. In this paper, we re-imagine benchmarking for robotic manipulation as state-of-the-art algorithmic implementations, alongside the usual set of tasks and experimental protocols. The added baseline implementations will provide a way to easily recreate SOTA numbers in a new local robotic setup, thus providing credible relative rankings between existing approaches and new work. However, these local rankings could vary between different setups. To resolve this issue, we build a mechanism for pooling experimental data between labs, and thus we establish a single global ranking for existing (and proposed) SOTA algorithms. Our benchmark, called Ranking-Based Robotics Benchmark (RB2), is evaluated on tasks that are inspired from clinically validated Southampton Hand Assessment Procedures. Our benchmark was run across two different labs and reveals several surprising findings. For example, extremely simple baselines like open-loop behavior cloning, outperform more complicated models (e.g. closed loop, RNN, Offline-RL, etc.) that are preferred by the field. We hope our fellow researchers will use RB2 to improve their research's quality and rigor.

RB2: Robotic Manipulation Benchmarking with a Twist

TL;DR

RB2 rethinks robotic manipulation benchmarking by coupling a fixed task suite with a model zoo of baselines and a central repository to enable local rankings that generalize into global rankings across labs. It introduces four SHAP-inspired tasks, five baseline methods, and a concrete protocol for data collection, training, and evaluation, enabling statistically grounded comparisons. Across two lab spaces, simple open-loop imitation often rivals or outperforms more complex methods like NDPs and MOReL, highlighting the gap between strong benchmarks and real-world performance. The global ranking mechanism democratizes benchmarking, allowing the community to collectively converge on credible baselines and robust progress in robotic manipulation.

Abstract

Benchmarks offer a scientific way to compare algorithms using objective performance metrics. Good benchmarks have two features: (a) they should be widely useful for many research groups; (b) and they should produce reproducible findings. In robotic manipulation research, there is a trade-off between reproducibility and broad accessibility. If the benchmark is kept restrictive (fixed hardware, objects), the numbers are reproducible but the setup becomes less general. On the other hand, a benchmark could be a loose set of protocols (e.g. object sets) but the underlying variation in setups make the results non-reproducible. In this paper, we re-imagine benchmarking for robotic manipulation as state-of-the-art algorithmic implementations, alongside the usual set of tasks and experimental protocols. The added baseline implementations will provide a way to easily recreate SOTA numbers in a new local robotic setup, thus providing credible relative rankings between existing approaches and new work. However, these local rankings could vary between different setups. To resolve this issue, we build a mechanism for pooling experimental data between labs, and thus we establish a single global ranking for existing (and proposed) SOTA algorithms. Our benchmark, called Ranking-Based Robotics Benchmark (RB2), is evaluated on tasks that are inspired from clinically validated Southampton Hand Assessment Procedures. Our benchmark was run across two different labs and reveals several surprising findings. For example, extremely simple baselines like open-loop behavior cloning, outperform more complicated models (e.g. closed loop, RNN, Offline-RL, etc.) that are preferred by the field. We hope our fellow researchers will use RB2 to improve their research's quality and rigor.
Paper Structure (36 sections, 3 figures, 1 table)

This paper contains 36 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Robotics Benchmarks
  4. Simulation Benchmarks
  5. Benchmarking with a Twist
  6. Benchmark Details - Tasks and Experimental Protocols
  7. Setup:
  8. Tasks ($\tau$):
  9. Task/Environment Parameters ($\theta$):
  10. Baselines ($\phi$):
  11. Local Ranking ($R$):
  12. Benchmark Instantiation: RB2
  13. Collection and Evaluation Protocols ($m$):
  14. Dataset Creation:
  15. Testing Procedure:
  16. ...and 21 more sections

Figures (3)

  • Figure 1: We present RB2, a real-world robot learning benchmark consisting of four manipulation tasks needed in daily human activity: pouring, scooping, zipping, and insertion. We provide experimentation, training and evaluation procedures as well as implementations of several state-of-the-art robot learning methods. Code, documentation and other details can be found at at: https://agi-labs.github.io/rb2/ .
  • Figure 2: We present four manipulation tasks: pouring, scooping, zipping and insertion as part of RB2. Each task involves a set of train (green) and (red) test objects.
  • Figure 3: Results of five baselines on four tasks. By definition, metrics are normalized to [0,1], w/ 1 being best possible performance. As the results show Open-Loop BC often outperforms NDP, and closed loop baselines. Note Offline RL is omitted since it never succeeds for any task.