Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GenDOM: Generalizable One-shot Deformable Object Manipulation with Parameter-Aware Policy

So Kuroki, Jiaxian Guo, Tatsuya Matsushima, Takuya Okubo, Masato Kobayashi, Yuya Ikeda, Ryosuke Takanami, Paul Yoo, Yutaka Matsuo, Yusuke Iwasawa

TL;DR

GenDOM tackles the problem of generalizing deformable-object manipulation with minimal real-world data by learning a parameter-conditioned policy that is trained across diverse simulated deformables using Young's modulus $p^y$ and Poisson's ratio $p^l$. At inference, a gradient-based Real2Sim procedure estimates these parameters from a single real demonstration by aligning point-cloud grid densities in a differentiable simulator, and these estimates condition the policy directly. Empirical results show substantial improvements in both simulation (ID 62% and OOD 15% over baselines) and real-world rope and cloth tasks, plus robust real-robot deployment advantages over data-intensive methods. The framework reduces data collection costs and enables cross-dynamics generalization, broadening the practical applicability of deformable-object manipulation.

Abstract

Due to the inherent uncertainty in their deformability during motion, previous methods in deformable object manipulation, such as rope and cloth, often required hundreds of real-world demonstrations to train a manipulation policy for each object, which hinders their applications in our ever-changing world. To address this issue, we introduce GenDOM, a framework that allows the manipulation policy to handle different deformable objects with only a single real-world demonstration. To achieve this, we augment the policy by conditioning it on deformable object parameters and training it with a diverse range of simulated deformable objects so that the policy can adjust actions based on different object parameters. At the time of inference, given a new object, GenDOM can estimate the deformable object parameters with only a single real-world demonstration by minimizing the disparity between the grid density of point clouds of real-world demonstrations and simulations in a differentiable physics simulator. Empirical validations on both simulated and real-world object manipulation setups clearly show that our method can manipulate different objects with a single demonstration and significantly outperforms the baseline in both environments (a 62% improvement for in-domain ropes and a 15% improvement for out-of-distribution ropes in simulation, as well as a 26% improvement for ropes and a 50% improvement for cloths in the real world), demonstrating the effectiveness of our approach in one-shot deformable object manipulation.

GenDOM: Generalizable One-shot Deformable Object Manipulation with Parameter-Aware Policy

TL;DR

GenDOM tackles the problem of generalizing deformable-object manipulation with minimal real-world data by learning a parameter-conditioned policy that is trained across diverse simulated deformables using Young's modulus and Poisson's ratio . At inference, a gradient-based Real2Sim procedure estimates these parameters from a single real demonstration by aligning point-cloud grid densities in a differentiable simulator, and these estimates condition the policy directly. Empirical results show substantial improvements in both simulation (ID 62% and OOD 15% over baselines) and real-world rope and cloth tasks, plus robust real-robot deployment advantages over data-intensive methods. The framework reduces data collection costs and enables cross-dynamics generalization, broadening the practical applicability of deformable-object manipulation.

Abstract

Due to the inherent uncertainty in their deformability during motion, previous methods in deformable object manipulation, such as rope and cloth, often required hundreds of real-world demonstrations to train a manipulation policy for each object, which hinders their applications in our ever-changing world. To address this issue, we introduce GenDOM, a framework that allows the manipulation policy to handle different deformable objects with only a single real-world demonstration. To achieve this, we augment the policy by conditioning it on deformable object parameters and training it with a diverse range of simulated deformable objects so that the policy can adjust actions based on different object parameters. At the time of inference, given a new object, GenDOM can estimate the deformable object parameters with only a single real-world demonstration by minimizing the disparity between the grid density of point clouds of real-world demonstrations and simulations in a differentiable physics simulator. Empirical validations on both simulated and real-world object manipulation setups clearly show that our method can manipulate different objects with a single demonstration and significantly outperforms the baseline in both environments (a 62% improvement for in-domain ropes and a 15% improvement for out-of-distribution ropes in simulation, as well as a 26% improvement for ropes and a 50% improvement for cloths in the real world), demonstrating the effectiveness of our approach in one-shot deformable object manipulation.
Paper Structure (20 sections, 1 equation, 3 figures, 7 tables)

This paper contains 20 sections, 1 equation, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deformable Object Manipulation
  4. Sim2Real
  5. Real2Sim
  6. Real2Sim2Real
  7. Proposed Method
  8. Task Description
  9. Training Policy with Diverse Parameters in Simulator
  10. Gradient-based Real2Sim Parameter Optimization
  11. Experiments
  12. Tasks
  13. Environments Setup
  14. Sim2Sim Performance
  15. Real2Sim Performance
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Task visualization. Rope reaching: A robot arm guides a rope end towards a release coordinate, after which the rope moves gravitationally towards goal coordinate $g$. Rope casting: A robot arm performs a momentary action in mid-air, guiding the rope towards coordinate ($g_x, g_y$). Cloth spreading: A robot arm handles a cloth's center and acts with acceleration to spread it over a table.
  • Figure 2: Overview of our Pipeline: (a) We randomly set parameters in a simulation to collect release and goal coordinates. Using this dataset, we trained a simple MLP policy. (b) For inference, parameters are optimized from real-world objects by comparing point cloud grid densities between simulation and reality through differentiable physics.
  • Figure 3: (a) Three different type ropes: cotton rope, polyester rope, and polyethylene rope. (b) Two different types of cloth: cotton cloth and rubber cloth. (c) Our experiments setup for Real2Sim: two Intel Realsense Depth Camera D435i and xArm 7.