Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Grasp to Act: Dexterous Grasping for Tool Use in Dynamic Settings

Harsh Gupta, Mohammad Amin Mirzaee, Wenzhen Yuan

TL;DR

This work presents Grasp-to-Act, a hybrid system that combines physics-based grasp optimization with reinforcement-learning-based grasp adaptation to maintain stable grasps throughout functional manipulation tasks, demonstrating stable functional grasps under dynamic, contact-rich conditions.

Abstract

Achieving robust grasping with dexterous hands remains challenging, especially when manipulation involves dynamic forces such as impacts, torques, and continuous resistance--situations common in real-world tool use. Existing methods largely optimize grasps for static geometric stability and often fail once external forces arise during manipulation. We present Grasp-to-Act, a hybrid system that combines physics-based grasp optimization with reinforcement-learning-based grasp adaptation to maintain stable grasps throughout functional manipulation tasks. Our method synthesizes robust grasp configurations informed by human demonstrations and employs an adaptive controller that residually issues joint corrections to prevent in-hand slip while tracking the object trajectory. Grasp-to-Act enables robust zero-shot sim-to-real transfer across five dynamic tool-use tasks--hammering, sawing, cutting, stirring, and scooping--consistently outperforming baselines. Across simulation and real-world hardware trials with a 16-DoF dexterous hand, our method reduces translational and rotational in-hand slip and achieves the highest task completion rates, demonstrating stable functional grasps under dynamic, contact-rich conditions.

Grasp to Act: Dexterous Grasping for Tool Use in Dynamic Settings

TL;DR

This work presents Grasp-to-Act, a hybrid system that combines physics-based grasp optimization with reinforcement-learning-based grasp adaptation to maintain stable grasps throughout functional manipulation tasks, demonstrating stable functional grasps under dynamic, contact-rich conditions.

Abstract

Achieving robust grasping with dexterous hands remains challenging, especially when manipulation involves dynamic forces such as impacts, torques, and continuous resistance--situations common in real-world tool use. Existing methods largely optimize grasps for static geometric stability and often fail once external forces arise during manipulation. We present Grasp-to-Act, a hybrid system that combines physics-based grasp optimization with reinforcement-learning-based grasp adaptation to maintain stable grasps throughout functional manipulation tasks. Our method synthesizes robust grasp configurations informed by human demonstrations and employs an adaptive controller that residually issues joint corrections to prevent in-hand slip while tracking the object trajectory. Grasp-to-Act enables robust zero-shot sim-to-real transfer across five dynamic tool-use tasks--hammering, sawing, cutting, stirring, and scooping--consistently outperforming baselines. Across simulation and real-world hardware trials with a 16-DoF dexterous hand, our method reduces translational and rotational in-hand slip and achieves the highest task completion rates, demonstrating stable functional grasps under dynamic, contact-rich conditions.
Paper Structure (29 sections, 2 equations, 6 figures, 2 tables)

This paper contains 29 sections, 2 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dextrous Grasp Synthesis
  4. Reinforcement Learning for Dexterous Manipulation
  5. Method
  6. Human Demonstration
  7. Grasp Synthesis and Evaluation
  8. Grasp Generation
  9. Functional Wrist Region Sampling
  10. Finger Configuration Sampling
  11. Grasp Stability Scoring
  12. Online Grasp Adaptation
  13. Simulation Environment
  14. Reinforcement Learning Formulation
  15. Experiments
  16. ...and 14 more sections

Figures (6)

  • Figure 1: Overview of the Grasp-to-Act framework. (Left) A human demonstration trajectory provides the reference of the dynamic conditions during tool using, as well as the initial grasping locations. Accordingly, we sample a series of grasps and score them to identify stable grasp candidates (middle left). A reinforcement learning policy then performs online-grasp adaptation by adjusting finger joint positions under task-specific force conditioning (middle right). The resulting policy is zero-shot deployed in the real world (right).
  • Figure 2: Pipeline of sampling and evaluating grasps. (A) We initialize the range of grasp locations based on human demonstration. (B-C) Candidate grasps are generated by varying joint-level parameters. (D) Each grasp is evaluated through wrench-space stability tests along six force and torque axes. (E) Top-ranked grasps are used as the initial grasp for conducting the tool-using trajectory, followed by online RL-based adaptation.
  • Figure 3: Real-world experiment setup. The workspace includes an (1) AprilTag, (2) tool tracking markers, (3) hand tracking markers, and an (4) RGBD camera.
  • Figure 4: Human demonstrations of the five functional tasks we test with in this paper: hammering, sawing, cutting, stirring, and scooping.
  • Figure 5: Real-world results across five functional tasks. Comparison of (a) In-hand translational slip distance $E_t$ (cm), (b) In-hand slip rotation distance $E_\theta$ (°), and (c) task completion $\mathcal{T}$. Task-specific definitions of $\mathcal{T}$ are provided in Section \ref{['sec:tasks']}.
  • ...and 1 more figures