DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects

TL;DR

This work tackles real-to-sim for highly deformable objects by learning physical parameters of a differentiable cloth simulator directly from real depth-point clouds. The authors introduce DiffCloud, an end-to-end pipeline that differentiates through both a point-cloud sampler and a mesh-based simulator to backpropagate a surface-alignment loss to material parameters such as mass and stiffness. DiffCloud achieves alignment with real objects faster than prior inverse-model baselines, reducing data collection and training requirements by orders of magnitude while maintaining or improving qualitative matching across tasks. The approach enables agile exploration of deformable manipulation and holds promise for broader, more interactive real-to-sim workflows in robotics and graphics.

Abstract

Research in manipulation of deformable objects is typically conducted on a limited range of scenarios, because handling each scenario on hardware takes significant effort. Realistic simulators with support for various types of deformations and interactions have the potential to speed up experimentation with novel tasks and algorithms. However, for highly deformable objects it is challenging to align the output of a simulator with the behavior of real objects. Manual tuning is not intuitive, hence automated methods are needed. We view this alignment problem as a joint perception-inference challenge and demonstrate how to use recent neural network architectures to successfully perform simulation parameter inference from real point clouds. We analyze the performance of various architectures, comparing their data and training requirements. Furthermore, we propose to leverage differentiable point cloud sampling and differentiable simulation to significantly reduce the time to achieve the alignment. We employ an efficient way to propagate gradients from point clouds to simulated meshes and further through to the physical simulation parameters, such as mass and stiffness. Experiments with highly deformable objects show that our method can achieve comparable or better alignment with real object behavior, while reducing the time needed to achieve this by more than an order of magnitude. Videos and supplementary material are available at https://diffcloud.github.io.

Figures (7)

• Figure 1: Experimental setup. We execute deformable manipulation trajectories using a Kinova Gen3 arm. We post-process observations recorded from two stereo depth cameras (Intel D435) to generate merged point clouds with the robot arm masked from view. These observations are fed to our proposed method DiffCloud for real-to-sim parameter estimation.
• Figure 2: Overview of DiffCloud: the proposed method for real-to-sim parameter estimation from point clouds. DiffCloud combines differentiable point cloud sampling with a differentiable mesh-based simulator to propagate losses computed between simulated (blue) and real (green, with red noise artifacts) point clouds to the underlying simulation properties. We visualize updating mass and stiffness of a simulated cloth lifted off of a table.
• Figure 3: Left: We visualize the average compute times across all methods for performing parameter estimation in the real lift and fold scenarios. Compared to the baseline inverse models, DiffCloud achieves more than an order of magnitude speedup, since it eliminates the need to pre-generate a dataset and train on it. Right: The optimized DiffCloud parameters found in the lift scenario correspond to the intuitive physical properties of real cloths, ranging from highly deformable to shape retaining. Each darkened circle represents the category median across three trajectories per cloth type.
• Figure 4: Left: A Kinova robot executes trajectories to lift real cloths from a flat starting state. From these trajectories, DiffCloud accurately infers stiffness and mass parameters capturing the observed degree of collapsibility in the real cloths. Right: Across all cloth types, DiffCloud achieves lower loss on average than all competing baselines on the lift scenario.
• Figure 5: Left: DiffCloud correctly learns to approximate low mass/high stiffness for the shape retaining cloth (1st row) such that three corners lift off the table mid-fold (2nd row, 2nd column), and high mass/low stiffness result for the heavy, highly deformable polka dot fabric (3rd row) such that all ungrasped corners rest on the table mid-fold (4th row, 2nd column). Right: Compared to data-driven baselines, DiffCloud achieves lower or comparable loss in 13/15 trajectories across categories. We note that all methods struggle to robustly estimate parameters for 2/15 paper towel (shape retaining) manipulation trajectories, which appear as outliers. This is due to difficulties perceiving very thin sheets in motion, which affects all methods.