A Benchmarking Study of Vision-Based Robotic Grasping Algorithms: A Comparative Analysis

TL;DR

This work addresses the lack of standardized benchmarking in vision-based robotic grasping by conducting a large, protocol-driven comparison of four algorithms (GG-CNN, a ResNet-based method, Top Surface, and Mask-based) using a common top-down perception setup. It combines real-world and simulation experiments across two laboratories with 10 YCB objects, varied lighting, textures, cameras, and grippers, and evaluates performance with Grasp Score and Grasp Success Score. A modular ROS-based benchmarking pipeline, open-source software, and detailed experimental configurations enable apples-to-apples comparisons and reproducibility, yielding insights into algorithm strengths, failure modes, and cross-lab repeatability. The findings indicate analytical methods often outperform data-driven ones across diverse conditions, though learning-based methods can excel in certain simulated environments, highlighting the importance of dataset design, sensor characteristics, and hardware in grasp synthesis research.

Abstract

We present a benchmarking study of vision-based robotic grasping algorithms and provide a comparative analysis. In particular, we compare two machine-learning-based and two analytical algorithms using an existing benchmarking protocol from the literature and determine the algorithms strengths and weaknesses under different experimental conditions. These conditions include variations in lighting, background textures, cameras with different noise levels, and grippers. We also run analogous experiments in simulations and with real robots and present the discrepancies. Some experiments are also run in two different laboratories using the same protocols to further analyze the repeatability of our results. We believe that this study, comprising 5040 experiments, provides important insights into the role and challenges of systematic experimentation in robotic manipulation and guides the development of new algorithms by considering the factors that could impact the performance. The experiment recordings and our benchmarking software are publicly available.

Figures (8)

• Figure 1: Illustrations of the flow of analytical algorithms. The image on top represents the mask-based algorithm, and the image on the bottom represents the top surface algorithm.
• Figure 2:
• Figure 3: The images on the left show the Franka Panda robot equipped with a Franka Gripper performing object-picking tasks in our lab at Worcester Polytechnic Institute (WPI), operating on a textured grasping surface under ambient lighting of 60 lux. The images on the right show a Universal Robot with a Robotiq Gripper picking objects in our lab at University of Massachusetts Lowell, operating on a non-textured grasping surface under ambient lighting of 340 lux.
• Figure 4: The YCB objects utilized for the benchmarking experiments.
• Figure 5: Grasps synthesized by different grasping algorithms for Medium Clamp in the simulation setup (Mask-based, ResNet, Top Surface, GG-CNN in order from top-left to bottom-right.)