Multi-fingered Dynamic Grasping for Unknown Objects
Yannick Burkhardt, Qian Feng, Jianxiang Feng, Karan Sharma, Zhaopeng Chen, Alois Knoll
TL;DR
The paper addresses dexterous grasping of unknown, dynamic objects using a five-finger hand by introducing a two-process framework: Target Model Generation builds an online internal model from RGB-D data, while Grasp Control selects and executes high-quality grasps via the generative FFHNet, incorporating velocity estimation to handle disturbances. The Target Model Generation pipeline uses a transformer-based tracker (TransT-M), ICP-based alignment, and point-cloud fusion with Basis Point Set encoding to maintain a robust 3D representation. The Grasp Control module employs a dynamic grasping metric, velocity-aware control, and a grasp-execution policy that can cope with temporary visual feedback loss, validated through real hardware experiments on conveyor belts and human-robot handovers. Results show a strong, practical performance (average success around 71.7% on moving objects, and up to 90% for certain speeds and cooperative handovers), demonstrating a significant step toward autonomous multi-fingered grasping of unknown objects in dynamic environments.
Abstract
Dexterous grasping of unseen objects in dynamic environments is an essential prerequisite for the advanced manipulation of autonomous robots. Prior advances rely on several assumptions that simplify the setup, including environment stationarity, pre-defined objects, and low-dimensional end-effectors. Though easing the problem and enabling progress, it undermined the complexity of the real world. Aiming to relax these assumptions, we present a dynamic grasping framework for unknown objects in this work, which uses a five-fingered hand with visual servo control and can compensate for external disturbances. To establish such a system on real hardware, we leverage the recent advances in real-time dexterous generative grasp synthesis and introduce several techniques to secure the robustness and performance of the overall system. Our experiments on real hardware verify the ability of the proposed system to reliably grasp unknown dynamic objects in two realistic scenarios: objects on a conveyor belt and human-robot handover. Note that there has been no prior work that can achieve dynamic multi-fingered grasping for unknown objects like ours up to the time of writing this paper. We hope our pioneering work in this direction can provide inspiration to the community and pave the way for further algorithmic and engineering advances on this challenging task. A video of the experiments is available at https://youtu.be/b87zGNoKELg.