Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabrics

Tyler Ga Wei Lum, Martin Matak, Viktor Makoviychuk, Ankur Handa, Arthur Allshire, Tucker Hermans, Nathan D. Ratliff, Karl Van Wyk

TL;DR

DextrAH-G tackles the challenge of fast, safe, and generalizable dexterous grasping by fusing a geometric fabric-based controller with reinforcement learning and teacher-student distillation. A privileged fabric-guided policy is trained in simulation to master grasping across 140 objects, and is distilled into a depth-based policy that operates in real time with depth images, enabling zero-shot sim-to-real transfer. Key innovations include a vectorized geometric fabric that enforces collision avoidance and joint constraints, and an 11-dimensional PCA-based action space for coordinated arm-hand control. Real-world tests with an Allegro hand and KUKA arm demonstrate strong performance (87% success, 5.63 picks/min) and robust behavior, illustrating practical potential for scalable, safe dexterous manipulation in clutter-free settings.

Abstract

A pivotal challenge in robotics is achieving fast, safe, and robust dexterous grasping across a diverse range of objects, an important goal within industrial applications. However, existing methods often have very limited speed, dexterity, and generality, along with limited or no hardware safety guarantees. In this work, we introduce DextrAH-G, a depth-based dexterous grasping policy trained entirely in simulation that combines reinforcement learning, geometric fabrics, and teacher-student distillation. We address key challenges in joint arm-hand policy learning, such as high-dimensional observation and action spaces, the sim2real gap, collision avoidance, and hardware constraints. DextrAH-G enables a 23 motor arm-hand robot to safely and continuously grasp and transport a large variety of objects at high speed using multi-modal inputs including depth images, allowing generalization across object geometry. Videos at https://sites.google.com/view/dextrah-g.

DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabrics

TL;DR

DextrAH-G tackles the challenge of fast, safe, and generalizable dexterous grasping by fusing a geometric fabric-based controller with reinforcement learning and teacher-student distillation. A privileged fabric-guided policy is trained in simulation to master grasping across 140 objects, and is distilled into a depth-based policy that operates in real time with depth images, enabling zero-shot sim-to-real transfer. Key innovations include a vectorized geometric fabric that enforces collision avoidance and joint constraints, and an 11-dimensional PCA-based action space for coordinated arm-hand control. Real-world tests with an Allegro hand and KUKA arm demonstrate strong performance (87% success, 5.63 picks/min) and robust behavior, illustrating practical potential for scalable, safe dexterous manipulation in clutter-free settings.

Abstract

A pivotal challenge in robotics is achieving fast, safe, and robust dexterous grasping across a diverse range of objects, an important goal within industrial applications. However, existing methods often have very limited speed, dexterity, and generality, along with limited or no hardware safety guarantees. In this work, we introduce DextrAH-G, a depth-based dexterous grasping policy trained entirely in simulation that combines reinforcement learning, geometric fabrics, and teacher-student distillation. We address key challenges in joint arm-hand policy learning, such as high-dimensional observation and action spaces, the sim2real gap, collision avoidance, and hardware constraints. DextrAH-G enables a 23 motor arm-hand robot to safely and continuously grasp and transport a large variety of objects at high speed using multi-modal inputs including depth images, allowing generalization across object geometry. Videos at https://sites.google.com/view/dextrah-g.
Paper Structure (43 sections, 6 equations, 20 figures, 3 tables)

This paper contains 43 sections, 6 equations, 20 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dexterous Grasping
  4. Policy Learning and Control
  5. DextrAH-G: Dexterous Arm-Hand Grasping
  6. Geometric Fabrics and Fabric-Guided Policies (FGPs)
  7. Teacher Privileged FGP Training (Reinforcement Learning)
  8. Student Depth FGP Training (Policy Distillation)
  9. Experiments
  10. Simulation
  11. Real-World
  12. Limitations
  13. Conclusion
  14. Additional Related Works on RL for Controlling Physical Robots
  15. Geometric Fabrics for Robot Control
  16. ...and 28 more sections

Figures (20)

  • Figure 1: DextrAH-G (Dexterous Arm-Hand Grasping) continuously controls a dexterous robot to grasp and transport a diverse range of objects directly from streaming depth images.
  • Figure 2: We train a privileged fabrics-guided policy (FGP) using RL (top), distill the privileged FGP into a depth FGP to predict the teacher's actions and object position (middle), and deploy DextrAH-G with a state machine in the real world for bin packing (bottom). See Table \ref{['tab:nomenclature']} for details.
  • Figure 3: The robot platform consists of an Allegro hand mounted to a Kuka LBR iiwa arm, one Intel Realsense D415 camera, a work table, and a bin to drop grasped objects.
  • Figure 4: DextrAH-G robustly grasps and transports diverse and novel objects in the real world.
  • Figure 5: We use a geometric fabric controller with integrated environment and self-collision avoidance. We visualize the geometric fabric's collision model, which models the robot as a set of spheres and the environment as a set of boxes.
  • ...and 15 more figures