Sim2Real Transfer for Vision-Based Grasp Verification

TL;DR

This work tackles verification of successful grasps for deformable objects using a vision-based approach that avoids force/tactile sensing. It introduces a two-stage GraspCheckNet system (YOLO-based gripper localization followed by a ResNet binary classifier) and a synthetic dataset, HSR-GraspSynth, to bridge the Sim2Real gap. A Visual Question Answering baseline is evaluated for zero-shot comparison. Real-world experiments on a Toyota HSR show strong gripper localization and object-presence detection, with on-device inference enabling low-latency integration into grasping pipelines; future work includes pipeline integration and domain adaptation.

Abstract

The verification of successful grasps is a crucial aspect of robot manipulation, particularly when handling deformable objects. Traditional methods relying on force and tactile sensors often struggle with deformable and non-rigid objects. In this work, we present a vision-based approach for grasp verification to determine whether the robotic gripper has successfully grasped an object. Our method employs a two-stage architecture; first YOLO-based object detection model to detect and locate the robot's gripper and then a ResNet-based classifier determines the presence of an object. To address the limitations of real-world data capture, we introduce HSR-GraspSynth, a synthetic dataset designed to simulate diverse grasping scenarios. Furthermore, we explore the use of Visual Question Answering capabilities as a zero-shot baseline to which we compare our model. Experimental results demonstrate that our approach achieves high accuracy in real-world environments, with potential for integration into grasping pipelines. Code and datasets are publicly available at https://github.com/pauamargant/HSR-GraspSynth .

Figures (4)

• Figure 1: Examples from the proposed dataset. The top row shows examples with an object within the gripper while the lower row corresponds to no object. Each example corresponds to a different batch.
• Figure 2: Illustration of the two-stage model architecture using object detection and image classification. The YOLO object detection model localizes the robot's gripper in the image, and the ResNet classification model uses the cropped image to determine whether there is an object in the gripper.
• Figure 3: Illustration of the clustering procedure applied to the detected bounding boxes. DBSCAN is used to identify clusters and assign a cluster label to each bounding box (left). Subsequently, the highest confidence bounding box from the cluster with the highest total confidence score is selected as the final detection (right).
• Figure 4: Sample detections using the object detection model. The red rectangle shows the detected bounding box. Top and lower rows show correct and incorrect detections respectively.