Quality-focused Active Adversarial Policy for Safe Grasping in Human-Robot Interaction

TL;DR

Safety in DNN-based vision-guided grasping is challenged when grasp quality maps assign high scores to the human hand or nearby objects in cluttered HRI. The authors present QFAAP, a benign active adversarial policy that combines Adversarial Quality Patch (AQP) and Projected Quality Gradient Descent (PQGD) to actively manipulate the grasp quality map so the hand is deprioritized (set to $Q$-zero) and nearby objects are suppressed, guiding the robot to safer grasps. AQP optimizes a patch via losses $L_q^p$, $L_{tv}$, and $L_d$ to maximize patch efficacy across images, while PQGD enables rapid hand-shape adaptation with a local, constrained update inside the hand region. Evaluated on three grasping datasets and a cobot, QFAAP improves safety metrics (e.g., ND-ACC up to 88% and CH-Rate down to 16%) and remains near real-time, highlighting a practical benign adversarial approach for safer HRI in cluttered environments.

Abstract

Vision-guided robot grasping methods based on Deep Neural Networks (DNNs) have achieved remarkable success in handling unknown objects, attributable to their powerful generalizability. However, these methods with this generalizability tend to recognize the human hand and its adjacent objects as graspable targets, compromising safety during Human-Robot Interaction (HRI). In this work, we propose the Quality-focused Active Adversarial Policy (QFAAP) to solve this problem. Specifically, the first part is the Adversarial Quality Patch (AQP), wherein we design the adversarial quality patch loss and leverage the grasp dataset to optimize a patch with high quality scores. Next, we construct the Projected Quality Gradient Descent (PQGD) and integrate it with the AQP, which contains only the hand region within each real-time frame, endowing the AQP with fast adaptability to the human hand shape. Through AQP and PQGD, the hand can be actively adversarial with the surrounding objects, lowering their quality scores. Therefore, further setting the quality score of the hand to zero will reduce the grasping priority of both the hand and its adjacent objects, enabling the robot to grasp other objects away from the hand without emergency stops. We conduct extensive experiments on the benchmark datasets and a cobot, showing the effectiveness of QFAAP. Our code and demo videos are available here: https://github.com/clee-jaist/QFAAP.

Figures (10)

• Figure 1: An example of a cluttered HRI scenario: the robot mistakenly identifies the human hand or adjacent objects as graspable targets for autonomous grasping, causing harm to the human. We highlight the robot, the human hand, and the target object using yellow, green, and blue borders, respectively.
• Figure 2: Pipeline of QFAAP: Firstly, the original RGB frame $\mathbf{x}$ is captured by the depth camera, and a hand segmentation algorithm (HS) is applied to obtain the hand mask $\mathcal{M}_h$. Next, the optimized AQP is incorporated into $\mathbf{x}$ while preserving only the hand region, generating $\mathbf{x}'$. In the third stage, PQGD is applied to $\mathbf{x}'$ with $\mathcal{M}_h$ to rapidly endorse the shape adaptability of AQP, producing $\mathbf{x}_t"$. In the fourth stage, $\mathbf{x}_t"$ is fed into the grasping model (GM) to obtain the quality map $\mathbf{Q}_t$, followed by getting the quality map $\tilde{\mathbf{Q}}_t^h$ outside the hand region by $\mathcal{M}_h$. Finally, selecting the optimal grasp (SOG) $g_t^*$ (emphasized by the green circle and orange dot) with the maximum quality score (emphasized by the white circle and orange dot) within $\tilde{\mathbf{Q}}_t^h$. The above process can effectively shift the initial hazardous grasp (the robot is emphasized as a blurred version) located near the hand (emphasized by the green line) toward a safer grasp (the object being grasped and the robot are emphasized with the blue and yellow borders.
• Figure 3: Experimental setup of robot grasping: primarily consisting of an Intel RealSense D435 depth camera, a UFactory 850 robot, a UFactory xArm gripper, and 20 novel objects (emphasized by blue borders).
• Figure 4: Line graphs showing the effectiveness of PQGD across all epochs, including its impact on the AQP optimized by GR-ConvNet and three different datasets, as well as the AQP optimized by SE-ResUNet and three different datasets. Here, the AQP and AQP&PQGD are represented by blue and purple lines, and we also use blue and purple dots to emphasize their corresponding maximum quality score across all epochs.
• Figure 5: Quality score visualization of AQP (first two rows) before and after adding PQGD (last two rows). Here, the GGCNN2 and the Cornell Grasp dataset are used to optimize the AQP. The AQP before and after adding PQGD are located in different locations. And AQP is scaled to 0.3 of the original size (the same size of the image).