Can Real-to-Sim Approaches Capture Dynamic Fabric Behavior for Robotic Fabric Manipulation?

TL;DR

The paper evaluates Real-to-Sim parameter estimation approaches for robotic fabric manipulation using two differentiable pipelines, a data-driven model, and a physics-informed neural network, across two simulators and five fabric types. It systematically tests training scenarios (lifting, wind, stretching) and unseen evaluation tasks (folding, fling, shaking) to assess generalization, finding that the simulation engine and real2sim method strongly influence performance while PINNs excel in quasi-static tasks but struggle dynamically. The study contributes a comprehensive comparison, a PINN-based real2sim approach, and a fabric manipulation dataset with synchronized RGB-D data, guiding future choices of simulators and estimation methods for fabric and garment manipulation. Overall, the results emphasize the need for accurate constitutive modeling and careful scenario design to reduce the Sim-to-Real gap in deformable object manipulation. The findings have practical implications for developing robust, generalizable fabric manipulation policies in robotics, particularly in selecting appropriate simulators and estimation strategies based on task dynamics.

Abstract

This paper presents a rigorous evaluation of Real-to-Sim parameter estimation approaches for fabric manipulation in robotics. The study systematically assesses three state-of-the-art approaches, namely two differential pipelines and a data-driven approach. We also devise a novel physics-informed neural network approach for physics parameter estimation. These approaches are interfaced with two simulations across multiple Real-to-Sim scenarios (lifting, wind blowing, and stretching) for five different fabric types and evaluated on three unseen scenarios (folding, fling, and shaking). We found that the simulation engines and the choice of Real-to-Sim approaches significantly impact fabric manipulation performance in our evaluation scenarios. Moreover, PINN observes superior performance in quasi-static tasks but shows limitations in dynamic scenarios.

Figures (4)

• Figure 1: This study focuses on real2sim physics parameter estimation for fabrics and evaluation across different scenarios. First,real-world fabric data is collected from three scenarios with five fabric types and 40 motion trajectories for real2sim. Physics parameters are then estimated using differentiable pipelines, deep learning, and physics-informed neural networks (PINNs) and applied within the same simulation engine. Evaluation is performed across three unseen scenarios with 120 motion trajectories, comparing simulated and real-world fabric behavior to assess accuracy and generalization. The study examines the impact of four real2sim approaches, two simulation engines, and three real2sim scenarios on estimation performance.
• Figure 2: The proposed pipeline consists of three stages: (1) Data Collection, (2) real2sim Physics Parameter Estimation of fabrics, and (3) Evaluation. In (1), real-world fabric behavior is recorded across six scenarios—three for real2sim (lifting, wind blowing, stretching) and three for evaluation (shaking, fling, folding). In (2), physics parameters are estimated using three approaches: differential pipelines (DiffCLOUD, DiffCP), deep learning (PhysNet duan2022learning), and physics-informed neural networks. Specifically, DiffSim DBLP:conf/icml/QiaoLKL20 is used to estimate stiffness and is then employed to simulate fabric behavior using the estimated values, while DiffTaichi DBLP:conf/iclr/HuALSCRD20 follows the same process for Young’s modulus and Poisson’s ratio. Finally, the evaluation stage assesses the generalization of estimated parameters by comparing simulated fabric behaviors to real-world observations across unseen scenarios.
• Figure 3: Visualization of DiffCloud and DiffCP approaches comparing fabric deformation states during lifting.
• Figure 4: Comparative visualization of fabric manipulation across different Real2Sim approaches and scenarios. The three rows represent different manipulation tasks: folding (top), fling (middle), and shaking (bottom). The rightmost columns show the real point cloud data and an overlaid comparison. Each approach is visualized using a distinct color for clear differentiation. The folding sequence shows the fabric's deformation from flat to folded states, while the fling and shaking sequence captures the dynamic motion of the fabric during the fling and shaking action.