FuseGrasp: Radar-Camera Fusion for Robotic Grasping of Transparent Objects

TL;DR

FuseGrasp addresses the poor depth perception and grasp reliability of RGB-D cameras for transparent objects by fusing mmWave radar with RGB-D through a SAR-based imaging pipeline and a four-module architecture. A two-stage training strategy leverages a large RGB-D dataset followed by fine-tuning on a small RGB-D-Radar set, enabling effective depth reconstruction and material identification to inform grasping. The approach achieves higher depth accuracy, robust material classification, and significantly improved real-world grasp success across lighting conditions, underscoring its practical viability for transparent-object manipulation. This work demonstrates that radar-assisted sensing can substantially enhance robot perception and manipulation in environments with challenging visual properties.

Abstract

Transparent objects are prevalent in everyday environments, but their distinct physical properties pose significant challenges for camera-guided robotic arms. Current research is mainly dependent on camera-only approaches, which often falter in suboptimal conditions, such as low-light environments. In response to this challenge, we present FuseGrasp, the first radar-camera fusion system tailored to enhance the transparent objects manipulation. FuseGrasp exploits the weak penetrating property of millimeter-wave (mmWave) signals, which causes transparent materials to appear opaque, and combines it with the precise motion control of a robotic arm to acquire high-quality mmWave radar images of transparent objects. The system employs a carefully designed deep neural network to fuse radar and camera imagery, thereby improving depth completion and elevating the success rate of object grasping. Nevertheless, training FuseGrasp effectively is non-trivial, due to limited radar image datasets for transparent objects. We address this issue utilizing large RGB-D dataset, and propose an effective two-stage training approach: we first pre-train FuseGrasp on a large public RGB-D dataset of transparent objects, then fine-tune it on a self-built small RGB-D-Radar dataset. Furthermore, as a byproduct, FuseGrasp can determine the composition of transparent objects, such as glass or plastic, leveraging the material identification capability of mmWave radar. This identification result facilitates the robotic arm in modulating its grip force appropriately. Extensive testing reveals that FuseGrasp significantly improves the accuracy of depth reconstruction and material identification for transparent objects. Moreover, real-world robotic trials have confirmed that FuseGrasp markedly enhances the handling of transparent items. A video demonstration of FuseGrasp is available at https://youtu.be/MWDqv0sRSok.

Figures (20)

• Figure 1: Top. Camera-based solutions using RGB-D imagery often face challenges in accurately detecting and grasping transparent objects under variable lighting conditions. Bottom. FuseGrasp strategically fuses RGB-D and radar imagery to enhance the precision and reliability of robotic grasping when handling transparent objects.
• Figure 2: Left. Physical antenna array layouts on mmWave radar boards, and the $2 \times 4$ Tx-Rx pairs used in FuseGrasp. Right. Sequence of transmitted signals (red) and received signals (blue).
• Figure 3: Formation of received signal. With the movement of the robotic arm, the mmWave radar attached to the hand will generate a virtual antenna array. By jointly processing the received signal at each antenna element, radar images of the detected targets can be synthesized.
• Figure 4: Visualization of the preliminary RMA radar imaging (right) of one cup under dark conditions and the corresponding RGB image (left) and depth image (middle).
• Figure 5: The architecture of FuseGrasp. It integrates an RGB-D camera and mmWave radar through four key modules: robotic imaging, depth reconstruction, material identification, and grasping.