Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Cross-hand Policies for High-DOF Reaching and Grasping

Qijin She, Shishun Zhang, Yunfan Ye, Ruizhen Hu, Kai Xu

TL;DR

The paper tackles cross-gripper generalization for high-DOF reaching and grasping by learning a gripper-agnostic policy that predicts displacements of semantic key points, coupled with a gripper-specific adaptation module that maps those displacements to joint commands. It introduces a gripper-agnostic state representation combining semantic key points with an Interaction Bisector Surface (IBS) and uses a transformer-based Unified Policy Model to fuse information across fingers, followed by a lightweight Adaptation Model to realize gripper actuation. The training employs a two-stage process (joint and transfer) within a Soft Actor-Critic framework, enabling transfer of grasp policies from a source gripper to multiple dexterous targets, demonstrated over several hands and object sets, including partial-observation scenarios. The results show that IBS-based representations and the transformer policy substantially improve cross-gripper transfer and final grasp quality, with the approach achieving robust performance across diverse grippers and maintaining real-time feasibility in many settings.

Abstract

Reaching-and-grasping is a fundamental skill for robotic manipulation, but existing methods usually train models on a specific gripper and cannot be reused on another gripper. In this paper, we propose a novel method that can learn a unified policy model that can be easily transferred to different dexterous grippers. Our method consists of two stages: a gripper-agnostic policy model that predicts the displacements of pre-defined key points on the gripper, and a gripper-specific adaptation model that translates these displacements into adjustments for controlling the grippers' joints. The gripper state and interactions with objects are captured at the finger level using robust geometric representations, integrated with a transformer-based network to address variations in gripper morphology and geometry. In the experiments, we evaluate our method on several dexterous grippers and diverse objects, and the result shows that our method significantly outperforms the baseline methods. Pioneering the transfer of grasp policies across dexterous grippers, our method effectively demonstrates its potential for learning generalizable and transferable manipulation skills for various robotic hands.

Learning Cross-hand Policies for High-DOF Reaching and Grasping

TL;DR

The paper tackles cross-gripper generalization for high-DOF reaching and grasping by learning a gripper-agnostic policy that predicts displacements of semantic key points, coupled with a gripper-specific adaptation module that maps those displacements to joint commands. It introduces a gripper-agnostic state representation combining semantic key points with an Interaction Bisector Surface (IBS) and uses a transformer-based Unified Policy Model to fuse information across fingers, followed by a lightweight Adaptation Model to realize gripper actuation. The training employs a two-stage process (joint and transfer) within a Soft Actor-Critic framework, enabling transfer of grasp policies from a source gripper to multiple dexterous targets, demonstrated over several hands and object sets, including partial-observation scenarios. The results show that IBS-based representations and the transformer policy substantially improve cross-gripper transfer and final grasp quality, with the approach achieving robust performance across diverse grippers and maintaining real-time feasibility in many settings.

Abstract

Reaching-and-grasping is a fundamental skill for robotic manipulation, but existing methods usually train models on a specific gripper and cannot be reused on another gripper. In this paper, we propose a novel method that can learn a unified policy model that can be easily transferred to different dexterous grippers. Our method consists of two stages: a gripper-agnostic policy model that predicts the displacements of pre-defined key points on the gripper, and a gripper-specific adaptation model that translates these displacements into adjustments for controlling the grippers' joints. The gripper state and interactions with objects are captured at the finger level using robust geometric representations, integrated with a transformer-based network to address variations in gripper morphology and geometry. In the experiments, we evaluate our method on several dexterous grippers and diverse objects, and the result shows that our method significantly outperforms the baseline methods. Pioneering the transfer of grasp policies across dexterous grippers, our method effectively demonstrates its potential for learning generalizable and transferable manipulation skills for various robotic hands.
Paper Structure (21 sections, 3 equations, 5 figures, 4 tables)

This paper contains 21 sections, 3 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Static Generalizable Grasp Synthesis
  4. Kinematic Motion Retargeting
  5. Dynamic Policy Transfer
  6. Methods
  7. Gripper-agnostic Feature Extraction
  8. Unified Policy Model
  9. Specific Adaptation Model
  10. Network Training
  11. Experiments
  12. Experiment Setting
  13. Ablation Studies on the Policy Model
  14. Effectiveness of two-stage hierarchical framework
  15. Importance of IBS
  16. ...and 6 more sections

Figures (5)

  • Figure 1: The overview of one step of our proposed framework. Given the context of the scene and the configuration of the gripper, our method initially extracts gripper-agnostic features. These features are uniformly sent to the policy model to predict gripper-agnostic point displacements, which are forwarded to the adaptation models of various grippers for precise gripper control.
  • Figure 2: The policy network of our method. The network includes three components : (a) finger-wise feature encoder; (b) a transform encoder fusing information among fingers and representations; (c) finger-wise and global.
  • Figure 3: The hands and their semantic key points. (a) Shadow; (b) Schunk; (c) Mano; (d) Rutgers; (e) Allegro.
  • Figure 4: The visual comparisons to motion retargeting method. Our method can adapt to the pose change of the object.
  • Figure 5: The visual results of our method on different grippers (up to down) and objects (left to right). For each case, we show the initial configurations of grippers and two sampled frames during the reaching process with the final grasping.