ASGrasp: Generalizable Transparent Object Reconstruction and 6-DoF Grasp Detection from RGB-D Active Stereo Camera

TL;DR

This work targets robust 6-DoF grasping of transparent and specular objects, where conventional depth sensing fails. It introduces ASGrasp, combining an RGB-D active stereo-based two-layer depth estimator for visible and occluded geometry with a GSNet-based grasp detector operating on complete point clouds. A large synthetic STD-GraspNet dataset with domain randomization enables strong sim-to-real transfer and superior performance, including a 90%+ success rate in both simulation and real-world tests. The results show ASGrasp outperforms state-of-the-art methods and even surpasses the performance upper bound set by perfect visible point clouds, highlighting the practical impact for cluttered scenes with challenging materials.

Abstract

In this paper, we tackle the problem of grasping transparent and specular objects. This issue holds importance, yet it remains unsolved within the field of robotics due to failure of recover their accurate geometry by depth cameras. For the first time, we propose ASGrasp, a 6-DoF grasp detection network that uses an RGB-D active stereo camera. ASGrasp utilizes a two-layer learning-based stereo network for the purpose of transparent object reconstruction, enabling material-agnostic object grasping in cluttered environments. In contrast to existing RGB-D based grasp detection methods, which heavily depend on depth restoration networks and the quality of depth maps generated by depth cameras, our system distinguishes itself by its ability to directly utilize raw IR and RGB images for transparent object geometry reconstruction. We create an extensive synthetic dataset through domain randomization, which is based on GraspNet-1Billion. Our experiments demonstrate that ASGrasp can achieve over 90% success rate for generalizable transparent object grasping in both simulation and the real via seamless sim-to-real transfer. Our method significantly outperforms SOTA networks and even surpasses the performance upper bound set by perfect visible point cloud inputs.Project page: https://pku-epic.github.io/ASGrasp

Figures (4)

• Figure 1: Overview of proposed method ASGrasp and the dataset. Our approach takes one RGB image and left-right infrared (IR) pair as inputs, predicts visible and invisible point cloud to generate grasp pose. We train the model using the DREDS dataset and the rendered dataset based on GraspNet-1Billion in complex scenes.
• Figure 2: The architecture of our proposed approach involves the extraction of features ${F_c}$, ${F_{ir}^{l}}$, and ${F_{ir}^{r}}$ from the RGB image ${I_c}$, the left IR image ${I_{ir}^{l}}$, and the right IR image ${I_{ir}^{r}}$. ${F_{ir}^{l}}$ and ${F_{ir}^{r}}$ are used to construct a correlation pyramid. This correlation pyramid, along with ${F_c}$, is fed into a GRU network for the prediction of a complete point cloud, which includes the first-layer depth (visible point cloud) and the second-layer depth (invisible point cloud). Subsequently, GSNet generates grasp poses based on the complete point cloud.
• Figure 3: Real world evaluation dataset
• Figure 4: Qualitative comparison of point cloud reconstruction for an exemplar test data from STD-GraspNet.