Adversarial Game-Theoretic Algorithm for Dexterous Grasp Synthesis

TL;DR

This work tackles robust dexterous grasp synthesis for multi-fingered hands by explicitly accounting for adversarial object motion. It introduces a two-player game where the robot hand (Player 1) seeks a feasible grasp while an adversary (Player 2) perturbs the object to escape, coupled by a differentiable firm grasp condition that ensures the object cannot move without collision. The approach uses a point-set representation with differentiable collision and self-collision constraints and solves the game via an iterative best-response with augmented Lagrangian optimization, achieving online performance and improved success rates over state-of-the-art baselines in simulation and real-robot experiments. Empirically, the method yields higher grasp success across dexterous hands and demonstrates robustness to extreme shape changes and real-world disturbances, highlighting its potential for reliable, real-time dexterous manipulation. The work points to future improvements including frictional effects, better initialization, and approaches to escape local minima in the inner optimizers.

Abstract

For many complex tasks, multi-finger robot hands are poised to revolutionize how we interact with the world, but reliably grasping objects remains a significant challenge. We focus on the problem of synthesizing grasps for multi-finger robot hands that, given a target object's geometry and pose, computes a hand configuration. Existing approaches often struggle to produce reliable grasps that sufficiently constrain object motion, leading to instability under disturbances and failed grasps. A key reason is that during grasp generation, they typically focus on resisting a single wrench, while ignoring the object's potential for adversarial movements, such as escaping. We propose a new grasp-synthesis approach that explicitly captures and leverages the adversarial object motion in grasp generation by formulating the problem as a two-player game. One player controls the robot to generate feasible grasp configurations, while the other adversarially controls the object to seek motions that attempt to escape from the grasp. Simulation experiments on various robot platforms and target objects show that our approach achieves a success rate of 75.78%, up to 19.61% higher than the state-of-the-art baseline. The two-player game mechanism improves the grasping success rate by 27.40% over the method without the game formulation. Our approach requires only 0.28-1.04 seconds on average to generate a grasp configuration, depending on the robot platform, making it suitable for real-world deployment. In real-world experiments, our approach achieves an average success rate of 85.0% on ShadowHand and 87.5% on LeapHand, which confirms its feasibility and effectiveness in real robot setups.

Figures (7)

• Figure 1: Our approach address the grasp synthesis problem for multi-fingered robotic hands by formulating it as a two-player game: Player 1 controls the hand to generate feasible grasp configurations, while Player 2 adversarially controls the object to attempt escape.
• Figure 2: Overview of the proposed grasp synthesis approach. a) Our method relies on a firm grasp condition that once satisfied, any non-zero object transformation will result in object-robot penetration. b) The problem is formulated as a two-player game. Player 1 seeks to satisfy the firm grasp condition, while Player 2 attempts to break it. The two players compete specifically on this condition, whereas all other constraints are isolated within their respective optimization problems. c) We model the hand by attaching spatial points to base and joint frames: joints are represented as spheres at joint centers, and links as ellipsoids with foci at their endpoints. d) The object is modeled as a set of points directly sampled from its point cloud. Penetration occurs if any of these points fall within the robot’s collision volume. e) For algorithm initialization, we align the palm frame $\mathcal{F}_R^\text{palm}$ and the object frame $\mathcal{F}_O$.
• Figure 3: Visualization of convergence behaviors under the iterative best-response strategy. a) The algorithm converges to a fixed equilibrium point, where the robot hand successfully constrains the motion of the object. b) In other cases, the player interactions are trapped in a cyclic pattern.
• Figure 4: Comparison of grasp success rates across three robotic hands using 58 objects from the CMapDataset full set. (a) Distribution of success rates for three methods on Barrett, Allegro, ShadowHand, and their overall average. (b) Proportion of objects in which each method achieves the highest success rate. (c) How success rates vary under increasing disturbances.
• Figure 5: Key frames in extreme-case robustness evaluation. ShadowHand maintains a stable grasp while the target object continuously morphs across distinct categories.