GraphGarment: Learning Garment Dynamics for Bimanual Cloth Manipulation Tasks

TL;DR

GraphGarment tackles deformable garment manipulation by learning garment dynamics with a graph neural network and correcting sim-to-real discrepancies with a residual model. The approach uses a graph-based representation of garment–robot interactions, combined with model-based action sampling to reach an optimal pre-hanging configuration, and augments learning with demonstration data for hanging via Dynamic Movement Primitives. In simulation, it outperforms baselines in state prediction and hanging success, while real-world experiments demonstrate robust sim-to-real transfer with a small additional error thanks to residual corrections. This work offers a practical, data-driven framework for reliable bimanual garment manipulation with potential impact on domestic, healthcare, and industrial robotics.

Abstract

Physical manipulation of garments is often crucial when performing fabric-related tasks, such as hanging garments. However, due to the deformable nature of fabrics, these operations remain a significant challenge for robots in household, healthcare, and industrial environments. In this paper, we propose GraphGarment, a novel approach that models garment dynamics based on robot control inputs and applies the learned dynamics model to facilitate garment manipulation tasks such as hanging. Specifically, we use graphs to represent the interactions between the robot end-effector and the garment. GraphGarment uses a graph neural network (GNN) to learn a dynamics model that can predict the next garment state given the current state and input action in simulation. To address the substantial sim-to-real gap, we propose a residual model that compensates for garment state prediction errors, thereby improving real-world performance. The garment dynamics model is then applied to a model-based action sampling strategy, where it is utilized to manipulate the garment to a reference pre-hanging configuration for garment-hanging tasks. We conducted four experiments using six types of garments to validate our approach in both simulation and real-world settings. In simulation experiments, GraphGarment achieves better garment state prediction performance, with a prediction error 0.46 cm lower than the best baseline. Our approach also demonstrates improved performance in the garment-hanging simulation experiment with enhancements of 12%, 24%, and 10%, respectively. Moreover, real-world robot experiments confirm the robustness of sim-to-real transfer, with an error increase of 0.17 cm compared to simulation results. Supplementary material is available at:https://sites.google.com/view/graphgarment.

Figures (9)

• Figure 1: Our proposed approach, GraphGarment, allows a robot to learn a garment dynamics model that can predict the next state of the garment given the current garment state and robot action input. This is then used for the cloth manipulation of hanging the garment on a clothes hanger.
• Figure 2: This figure demonstrates the overview of GraphGarment: A GNN predicts garment dynamics in simulation, refined by a residual model with real-world data. Using this predictive model, a model-based action sampling strategy adjusts the garment for optimal pre-hanging. A bimanual robotic hanging experiment validates the approach in both simulation and reality.
• Figure 3: Left: The left figure shows the experiment's target garments, including simulated (Garment 1, Garment 2, Garment 3) and real garment (Garment 1, Garment 2, Garment 3). Right: The right figure displays a sample graph illustrating its construction.
• Figure 4: Residual Network: A PointNet-based residual network is utilized here to generate an offset value for each predicted point to correct the output of the dynamics model.
• Figure 5: This figure shows our proposed approach for predicting the next state of the garment. We demonstrate the garment states at timesteps $s_t$ and $s_{t+1}$. We also present the prediction result, along with the Chamfer distance computed between the predicted state $y_{t+1}$. and the actual state $s_{t+1}$. Additional demonstrations are available in the multimedia resource.