Unknown Object Grasping for Assistive Robotics

TL;DR

The paper addresses unknown-object grasping in assistive robotics by introducing a human-in-the-loop pipeline that couples stereo-based perception and shape completion with user-guided end-effector planning on a virtual hemisphere. A GraspIt!/PyBullet-based physics planner evaluates local grasps on a completed mesh, selecting the pose with the grasp-quality measure $\epsilon$ for execution under shared control. Validation on the DLR EDAN platform with the CLASH hand achieves an 87% grasp success rate across 10 unknown singulated objects and demonstrates scalability to clutter and shelf scenarios, while autonomy modes help reduce cognitive load compared to fully manual control. The approach is end-effector agnostic, customizable, and scalable to more complex scenes, offering a practical pathway for home-use assistive grasping with user-centric control and robust perception.

Abstract

We propose a novel pipeline for unknown object grasping in shared robotic autonomy scenarios. State-of-the-art methods for fully autonomous scenarios are typically learning-based approaches optimised for a specific end-effector, that generate grasp poses directly from sensor input. In the domain of assistive robotics, we seek instead to utilise the user's cognitive abilities for enhanced satisfaction, grasping performance, and alignment with their high level task-specific goals. Given a pair of stereo images, we perform unknown object instance segmentation and generate a 3D reconstruction of the object of interest. In shared control, the user then guides the robot end-effector across a virtual hemisphere centered around the object to their desired approach direction. A physics-based grasp planner finds the most stable local grasp on the reconstruction, and finally the user is guided by shared control to this grasp. In experiments on the DLR EDAN platform, we report a grasp success rate of 87% for 10 unknown objects, and demonstrate the method's capability to grasp objects in structured clutter and from shelves.

Figures (7)

• Figure 1: Grasping an unknown object in structured clutter with the DLR EDAN assistive robotic system. After selecting an object, the user guides the end-effector across a virtual hemisphere to their desired approach direction. Depicted is an author, not a target user.
• Figure 2: Our unknown object grasping pipeline. The perception module transforms a pair of rectified stereo images to the completed mesh of an unknown object. Unknown object instance segmentation is performed using INSTR instr, and for shape completion we use the method presented in humt2023_shape which applies a yan2022shapeformer trained on simulated Kinect data. In shared control sct, the user guides the end-effector across a virtual hemisphere centered around the unknown object, selects their desired approach position, and a physics-based grasp planning simulation finds the best local grasp at which the actual grasp is executed automatically.
• Figure 3: Grasp approach hemispheres, decided by object height. The power grasp allows for side-on grasps, and the precision grasp allows for greater sampling around the center point.
• Figure 4: Visualisation of GraspIt! simulation output for a soup can. From the user's selected approach (top-down), local approach poses are sampled from 3 circumferences spaced between 0-10$\degree$ from the input (blue crosses) projected onto the hemisphere. The hand approaches the soup from these poses but will keep moving until 3 contacts are reached, causing it to envelope around the object. Successful grasps are shown in circles, where the circle size is proportional to finger flexion (i.e. the smallest circles correspond to finger flexion 0.35, largest 0.1), and colour shows grasp quality.
• Figure 5: Shelf grasping experiment. 1. INSTR prediction 2. Completion prediction 3. User's selected hemisphere approach 4. GraspIt! predicted grasp 5. Successful grasp. More complex grasping scenarios utilise the user's intuition for collision avoidance.