Developing Modular Grasping and Manipulation Pipeline Infrastructure to Streamline Performance Benchmarking

TL;DR

The paper tackles the fragmentation and reproducibility challenges in robot manipulation benchmarking by proposing a modular grasping and manipulation pipeline infrastructure within the COMPARE Ecosystem. It introduces a state-machine workflow with nested behavior trees that enforces interchangeable component interfaces, enabling hardware-agnostic experiments and a path toward standardization between perception, grasp planning, motion planning, and control modules. Through a sequence of experiments—GPD evaluation across multiple robots, automated reset test apparatuses for large-scale testing, and integration of multiple vision-based grasping algorithms—the authors validate the architecture and demonstrate practical benchmarking capabilities, including a Docker-based reproducibility setup. The contributions offer architecture-level guidance for modularity and interoperability, with concrete plans for expanding component ecosystems and community-driven benchmarking standards that can accelerate open-source collaboration and cross-lab validation in robotic manipulation.

Abstract

The robot manipulation ecosystem currently faces issues with integrating open-source components and reproducing results. This limits the ability of the community to benchmark and compare the performance of different solutions to one another in an effective manner, instead relying on largely holistic evaluations. As part of the COMPARE Ecosystem project, we are developing modular grasping and manipulation pipeline infrastructure in order to streamline performance benchmarking. The infrastructure will be used towards the establishment of standards and guidelines for modularity and improved open-source development and benchmarking. This paper provides a high-level overview of the architecture of the pipeline infrastructure, experiments conducted to exercise it during development, and future work to expand its modularity.

Figures (1)

• Figure 1: Experiments performed during development.