Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Neural Descriptor Fields for Learning Contact-Aware Dynamic Recovery

Fan Yang, Zixuan Huang, Abhinav Kumar, Sergio Aguilera Marinovic, Soshi Iba, Rana Soltani Zarrin, Dmitry Berenson

TL;DR

This work tackles recovery from catastrophic drops in dexterous manipulation by learning a dynamic catching policy that also resets the robot to a state favorable for resuming the primary task. It introduces CADRE, which integrates pretrained Neural Descriptor Field–based implicit contact features into a PPO-trained recovery policy to reason about finger–object contact across varying geometries. Empirical results in simulation and hardware show CADRE outperforms pose- and point-cloud–based baselines on both seen and unseen objects, with improved recovery quality and downstream task performance, though sim-to-real gaps remain. The approach demonstrates the potential of incorporating implicit contact representations to enable robust, geometry-aware recovery in contact-rich manipulation with practical implications for robotic manipulation in unstructured environments.

Abstract

Real-world dexterous manipulation often encounters unexpected errors and disturbances, which can lead to catastrophic failures, such as dropping the manipulated object. To address this challenge, we focus on the problem of catching a falling object while it remains within grasping range and, importantly, resetting the system to a configuration favorable for resuming the primary manipulation task. We propose Contact-Aware Dynamic Recovery (CADRE), a reinforcement learning framework that incorporates a Neural Descriptor Field (NDF)-inspired module to extract implicit contact features. Compared to methods that rely solely on object pose or point cloud input, NDFs can directly reason about finger-object correspondence and adapt to different object geometries. Our experiments show that incorporating contact features improves training efficiency, enhances convergence performance for RL training, and ultimately leads to more successful recoveries. Additionally, we demonstrate that CADRE can generalize zero-shot to unseen objects with different geometries.

Leveraging Neural Descriptor Fields for Learning Contact-Aware Dynamic Recovery

TL;DR

This work tackles recovery from catastrophic drops in dexterous manipulation by learning a dynamic catching policy that also resets the robot to a state favorable for resuming the primary task. It introduces CADRE, which integrates pretrained Neural Descriptor Field–based implicit contact features into a PPO-trained recovery policy to reason about finger–object contact across varying geometries. Empirical results in simulation and hardware show CADRE outperforms pose- and point-cloud–based baselines on both seen and unseen objects, with improved recovery quality and downstream task performance, though sim-to-real gaps remain. The approach demonstrates the potential of incorporating implicit contact representations to enable robust, geometry-aware recovery in contact-rich manipulation with practical implications for robotic manipulation in unstructured environments.

Abstract

Real-world dexterous manipulation often encounters unexpected errors and disturbances, which can lead to catastrophic failures, such as dropping the manipulated object. To address this challenge, we focus on the problem of catching a falling object while it remains within grasping range and, importantly, resetting the system to a configuration favorable for resuming the primary manipulation task. We propose Contact-Aware Dynamic Recovery (CADRE), a reinforcement learning framework that incorporates a Neural Descriptor Field (NDF)-inspired module to extract implicit contact features. Compared to methods that rely solely on object pose or point cloud input, NDFs can directly reason about finger-object correspondence and adapt to different object geometries. Our experiments show that incorporating contact features improves training efficiency, enhances convergence performance for RL training, and ultimately leads to more successful recoveries. Additionally, we demonstrate that CADRE can generalize zero-shot to unseen objects with different geometries.
Paper Structure (21 sections, 4 equations, 3 figures, 3 tables)

This paper contains 21 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dynamic Manipulation
  4. Representation Learning for Manipulation
  5. Problem Statement
  6. Methods
  7. Preliminary: Neural Descriptor Fields
  8. Contact-Aware Dynamic Recovery
  9. Contact-Aware Grasp Feature
  10. Reinforcement Learning for Dynamic Recovery
  11. Experiments
  12. Simulation Experiment Setup
  13. Screwdriver Recovery
  14. Nut Recovery
  15. Baselines and Ablation
  16. ...and 6 more sections

Figures (3)

  • Figure 1: CADRE recovers from object-dropping failures by catching the object and resetting to states favorable for resuming the primary manipulation task. Leveraging contact information from a pre-trained NDF model, it generalizes recovery behaviors to objects with different geometries.
  • Figure 2: Contact-Aware Dynamic Recovery (CADRE) aims to catch a manipulated object when it falls from the grasp and recover to states that support the resumption of the manipulation task. CADRE leverages a pretrained NDF model to extract implicit contact features from the grasp. The contact-aware grasp features are incorporated into the RL observations, enhancing the policy's awareness of contact and improving recovery performance.
  • Figure 3: Fig. (a) and (b) show the environment step. Three different seeds are used for RL training and the average results are shown in Fig. (c) and (d). The variance is relatively small as we use a large batch size during training.