QBIT: Quality-Aware Cloud-Based Benchmarking for Robotic Insertion Tasks

TL;DR

QBIT addresses the need for quality-aware benchmarking in robotic insertion by introducing a cloud-based framework that extends beyond success rate with metrics such as force energy and force smoothness, while accounting for perception uncertainty and sim-to-real variability. It combines a microservice, containerized architecture with Kubernetes for large-scale simulations and ROS2-based interfaces for real robots, enabling seamless integration of new robots and task objects. The methodology includes randomized contact parameters in MuJoCo, a sphere-based contact decomposition for realistic simulation, and three insertion approaches (position-based, force-based, learning-based) to highlight metric-driven distinctions. The results show that QBIT can differentiate insertion strategies, reduce the sim-to-real gap, and accelerate translation from lab experiments to real-world industrial use, with code available on GitHub.

Abstract

Insertion tasks are fundamental yet challenging for robots, particularly in autonomous operations, due to their continuous interaction with the environment. AI-based approaches appear to be up to the challenge, but in production they must not only achieve high success rates. They must also ensure insertion quality and reliability. To address this, we introduce QBIT, a quality-aware benchmarking framework that incorporates additional metrics such as force energy, force smoothness and completion time to provide a comprehensive assessment. To ensure statistical significance and minimize the sim-to-real gap, we randomize contact parameters in the MuJoCo simulator, account for perceptual uncertainty, and conduct large-scale experiments on a Kubernetes-based infrastructure. Our microservice-oriented architecture ensures extensibility, broad applicability, and improved reproducibility. To facilitate seamless transitions to physical robotic testing, we use ROS2 with containerization to reduce integration barriers. We evaluate QBIT using three insertion approaches: geometricbased, force-based, and learning-based, in both simulated and real-world environments. In simulation, we compare the accuracy of contact simulation using different mesh decomposition techniques. Our results demonstrate the effectiveness of QBIT in comparing different insertion approaches and accelerating the transition from laboratory to real-world applications. Code is available on GitHub.

Figures (11)

• Figure 1: Overview of the QBIT benchmarking framework. It aims to use cloud computing and quality-aware metrics to evaluate and validate the insertion algorithms for with different task objects on different robots.
• Figure 2: Insertion task formulation. Left: Yellow plane represents force signal energy metric $E_{xy}$ orthogonal to the insertion direction $E_z$. Center: Start pose $H_s$ and target pose $H_t$ and hole with surface roughness. Right: Resulting metrics of different approaches solving insertion task on a real robot.
• Figure 3: Overview of the benchmark software architecture.
• Figure 4: Asymmetrically scale the number of tested algorithm instances and simulation instances to speed up experiments.
• Figure 5: Experimental setup. Left: Real robot system with peg in hole task, Center: Simulation environment and Right: Real robot system with electrical connectors task. Interface is extensible to different robots and task objects.