Learning a General Model: Folding Clothing with Topological Dynamics

TL;DR

This work tackles folding of complex garments under self-occlusion by introducing a general topological dynamics framework. It defines a low-dimensional topological graph G_t derived from occlusion-aware keypoints and region contours, and trains GNN-based forward and corrective dynamics to predict garment deformation, yielding a deformation Jacobian J_f for Jacobi-based control. The approach reduces state dimensionality, handles varying clothing configurations, and demonstrates jacket folding under self-occlusion in both simulated and real-world experiments with robust graph extraction and planning. The results suggest a scalable, category-agnostic pathway for deformable garment manipulation with practical implications for household robotics and assistive automation.

Abstract

The high degrees of freedom and complex structure of garments present significant challenges for clothing manipulation. In this paper, we propose a general topological dynamics model to fold complex clothing. By utilizing the visible folding structure as the topological skeleton, we design a novel topological graph to represent the clothing state. This topological graph is low-dimensional and applied for complex clothing in various folding states. It indicates the constraints of clothing and enables predictions regarding clothing movement. To extract graphs from self-occlusion, we apply semantic segmentation to analyze the occlusion relationships and decompose the clothing structure. The decomposed structure is then combined with keypoint detection to generate the topological graph. To analyze the behavior of the topological graph, we employ an improved Graph Neural Network (GNN) to learn the general dynamics. The GNN model can predict the deformation of clothing and is employed to calculate the deformation Jacobi matrix for control. Experiments using jackets validate the algorithm's effectiveness to recognize and fold complex clothing with self-occlusion.

Figures (7)

• Figure 1: We propose a new topology-based representation of clothing states. By characterizing clothing occlusions and keypoints, this representation can be extracted and generated from complex clothing in different folding states. A general GNN is applied to obtain the dynamics model of the topological graph.
• Figure 2: Overview. (a) In the workspace, an RGB image observation $\boldsymbol{o}_t$ is captured by the camera. (b) Semantic keypoints and occlusive layer semantic segmentation are extracted from the observation and matched to construct the topological graph, $G_t$. (c) During manipulation, the tracked keypoints are utilized to refine $G_t$ via a GNN-based corrective model, while the new state, $G_{t+1}$ is predicted by a GNN-based predictive model. The corrective model (d) and the predictive model (e) employ the message passing mechanism to fit the expected process. The DJM is obtained from the predictive model by back propagation. (f) Based on the topological state, the folding trajectory is generated in 4 stages: approaching, lifting, placing, and leaving. The robot actions, $\boldsymbol{a}_t$ are calculated by the Jacobian controller to follow the trajectory.
• Figure 3: The process to construct the topological graph. (a) The observation. (b) The keypoints. (c) The segmentation of occlusive regions. (d) The contours of connected regions. (e) The contours matched with keypoints. (f) The result.
• Figure 4: The process of one-step message passing in our graph net. (a) is the latent graph state to update. (b) is implemented first to obtain the regional feature $^{k+1}\boldsymbol{r}'_{\Lambda_i}$. (c) and (d) are calculated together then to update the node and edge states.
• Figure 5: The refined trajectory in lifting stage. The dashed green arc is the initial trajectory. The solid green arc is the moved trajectory. The blue arc is the arc crossing $\hat{\boldsymbol{p}}_f$ and $\hat{\boldsymbol{p}}_p$. The red arc is the interpolated refined trajectory.