Differentiable Particles for General-Purpose Deformable Object Manipulation

TL;DR

DiPac presents a general framework for deformable object manipulation by representing objects as differentiable particles and learning a calibrated MLS-MPM-based dynamics model. It blends learning, planning, and differentiable trajectory optimization in a trajectory-tree planner, enabling end-to-end gradient-based action selection from RGBD inputs. The key contributions include a differentiable calibration of dynamics to reduce sim-to-real gaps, a policy-guided trajectory tree planner, and strong empirical results across rope, beans, cloth, and liquids with robust transfer to new dynamics. The approach advances generality and data efficiency in deformable object manipulation and shows promise for real-world deployment and broader applicability.

Abstract

Deformable object manipulation is a long-standing challenge in robotics. While existing approaches often focus narrowly on a specific type of object, we seek a general-purpose algorithm, capable of manipulating many different types of objects: beans, rope, cloth, liquid, . . . . One key difficulty is a suitable representation, rich enough to capture object shape, dynamics for manipulation and yet simple enough to be acquired effectively from sensor data. Specifically, we propose Differentiable Particles (DiPac), a new algorithm for deformable object manipulation. DiPac represents a deformable object as a set of particles and uses a differentiable particle dynamics simulator to reason about robot manipulation. To find the best manipulation action, DiPac combines learning, planning, and trajectory optimization through differentiable trajectory tree optimization. Differentiable dynamics provides significant benefits and enable DiPac to (i) estimate the dynamics parameters efficiently, thereby narrowing the sim-to-real gap, and (ii) choose the best action by backpropagating the gradient along sampled trajectories. Both simulation and real-robot experiments show promising results. DiPac handles a variety of object types. By combining planning and learning, DiPac outperforms both pure model-based planning methods and pure data-driven learning methods. In addition, DiPac is robust and adapts to changes in dynamics, thereby enabling the transfer of an expert policy from one object to another with different physical properties, e.g., from a rigid rod to a deformable rope.

Figures (9)

• Figure 1: Deformable object manipulation tasks used in evaluation: the first three on a real robot and the last two in simulation. (a) Straighten the rope via pushing. (b) Sweep the scattered beans into a tray. (c) Hang a piece of cloth on a rack. (d) Pour liquid into a bowl. (e) Pour soup, a liquid-solid mixture, into a bowl. The top row shows the robot in action. The bottom row shows the corresponding particle representations.
• Figure 2: Model calibration. The gradient-based optimization updates the parameter set $\varphi$ to minimize the Chamfer distance between the predicted state $\hat{x}_t$ and the observed particle state $x_t$.
• Figure 3: Action selection via differentiable trajectory tree optimization. The search for the best action is guided by a learned initial policy $\pi_\theta$, together with random exploration policy $\pi_{\text{rand}}$. After rollouts, DiPac minimizes the total trajectory cost by back-propagating the gradient along each sampled trajectory.
• Figure 4: Real-robot experiments setup. The top row shows robot actions. The bottom row shows the desired goal states.
• Figure 5: Real robot deformable object manipulation task results on three different materials. We report the averaged Chamfer distance (meters) over steps.