SCENEREPLICA: Benchmarking Real-World Robot Manipulation by Creating Replicable Scenes

TL;DR

A new reproducible benchmark for evaluating robot manipulation in the real world, specifically focusing on a pick-and-place task, is presented, where representative algorithms are evaluated for object perception, grasping planning, and motion planning.

Abstract

We present a new reproducible benchmark for evaluating robot manipulation in the real world, specifically focusing on pick-and-place. Our benchmark uses the YCB objects, a commonly used dataset in the robotics community, to ensure that our results are comparable to other studies. Additionally, the benchmark is designed to be easily reproducible in the real world, making it accessible to researchers and practitioners. We also provide our experimental results and analyzes for model-based and model-free 6D robotic grasping on the benchmark, where representative algorithms are evaluated for object perception, grasping planning, and motion planning. We believe that our benchmark will be a valuable tool for advancing the field of robot manipulation. By providing a standardized evaluation framework, researchers can more easily compare different techniques and algorithms, leading to faster progress in developing robot manipulation methods.

Figures (8)

• Figure 1: 16 YCB objects are used in SceneReplica.
• Figure 2: (a,b) Illustration of motion planning check to filter out reachable locations. (c) Blue cubes remaining on table after planning indicate reachable locations. (d) The algorithm can be extended to different robots.
• Figure 3: 20 scenes in our SceneReplica benchmark with 5 YCB objects in each scene.
• Figure 4: The process of replicating a scene in the real world. The image of the reference scene is overlaid on the image of the real camera to guide how to place objects into the real-world scene.
• Figure 5: Illustration of the coordinate frames.