OmniDexGrasp: Generalizable Dexterous Grasping via Foundation Model and Force Feedback

TL;DR

OmniDexGrasp addresses the challenge of generalizable dexterous grasping by tying foundation-model-driven perception to a transfer-based action synthesis and force-aware control. It introduces three core components: (1) grasp image generation by foundation models to capture omni-prompts, (2) a human-image-to-robot-action transfer that reconstructs hands and objects and retargets to dexterous hands, and (3) a force-aware adaptive grasping scheme that uses force feedback for robust execution. The approach achieves high performance across six real-world grasp tasks and diverse objects, demonstrates strong generalization to unseen categories, and extends naturally to dexterous manipulation and cross-embodiment scenarios. This work highlights the potential of foundation-model-driven embodied intelligence for broadening robot capabilities without robot-specific data collection.

Abstract

Enabling robots to dexterously grasp and manipulate objects based on human commands is a promising direction in robotics. However, existing approaches are challenging to generalize across diverse objects or tasks due to the limited scale of semantic dexterous grasp datasets. Foundation models offer a new way to enhance generalization, yet directly leveraging them to generate feasible robotic actions remains challenging due to the gap between abstract model knowledge and physical robot execution. To address these challenges, we propose OmniDexGrasp, a generalizable framework that achieves omni-capabilities in user prompting, dexterous embodiment, and grasping tasks by combining foundation models with the transfer and control strategies. OmniDexGrasp integrates three key modules: (i) foundation models are used to enhance generalization by generating human grasp images supporting omni-capability of user prompt and task; (ii) a human-image-to-robot-action transfer strategy converts human demonstrations into executable robot actions, enabling omni dexterous embodiment; (iii) force-aware adaptive grasp strategy ensures robust and stable grasp execution. Experiments in simulation and on real robots validate the effectiveness of OmniDexGrasp on diverse user prompts, grasp task and dexterous hands, and further results show its extensibility to dexterous manipulation tasks.

Figures (7)

• Figure 1: OmniDexGrasp can achieve generalizable dexterous grasping with omni capabilities in user prompting, dexterous embodiment, scenes, and grasping tasks, by leveraging the foundation model and the propose transfer and grasp strategy.
• Figure 2: Overview of OmniDexGrasp: (1) human grasp image generation via foundation generative models, (2) human-to-robot action transfer mapping grasp images to dexterous robot actions, and (3) force-aware adaptive grasping for stable grasps.
• Figure 3: Visualization of the inputs and outputs of foundation generative models, illustrating the potential change of object pose in the generated image.
• Figure 4: Visualization of real world hardware platform and the objects used in experiments.
• Figure 5: Visualization of the six tasks (upper) and the whole grasping motion process (lower).