FoldNet: Learning Generalizable Closed-Loop Policy for Garment Folding via Keypoint-Driven Asset and Demonstration Synthesis

TL;DR

FoldNet tackles the data bottleneck in robotic garment folding by generating a large, keypoint-annotated asset library and high-quality folding demonstrations, augmented with a keypoint-guided error-recovery strategy (KG-DAgger). The framework yields an end-to-end diffusion-based policy trained in simulation that transfers to the real world and generalizes to unseen garments. Key contributions include a scalable garment mesh synthesis pipeline with automatic keypoint annotations and a data-augmentation strategy that substantially improves real-world folding success (up to 75% from 50%). The work demonstrates promising sim-to-real transfer and opens avenues for combining synthetic assets with recovery-driven imitation learning to robustly manipulate deformable objects.

Abstract

Due to the deformability of garments, generating a large amount of high-quality data for robotic garment manipulation tasks is highly challenging. In this paper, we present a synthetic garment dataset that can be used for robotic garment folding. We begin by constructing geometric garment templates based on keypoints and applying generative models to generate realistic texture patterns. Leveraging these keypoint annotations, we generate folding demonstrations in simulation and train folding policies via closed-loop imitation learning. To improve robustness, we propose KG-DAgger, which uses a keypoint-based strategy to generate demonstration data for recovering from failures. KG-DAgger significantly improves the model performance, boosting the real-world success rate by 25\%. After training with 15K trajectories (about 2M image-action pairs), the model achieves a 75\% success rate in the real world. Experiments in both simulation and real-world settings validate the effectiveness of our proposed framework.

Figures (13)

• Figure 1: FoldNet is a dataset designed for robotic garment manipulation. It provides (1) a large collection of synthetic garment assets with keypoint annotations, (2) high-quality folding demonstrations, and (3) keypoint-based error recovery demonstrations. Leveraging these assets, FoldNet supports a wide range of downstream tasks, including (a) keypoint detection, (b) folding in simulation, and (c) folding in the real world.
• Figure 2: Pipeline for garment mesh synthesis. By performing geometry generation, texture generation, combining-and-filtering, we can synthesize scalable, high-quality garment meshes.
• Figure 3: Synthetic garment meshes. These static garment meshes can be used for subsequent physics simulation and policy learning.
• Figure 4: Keypoint-based demonstration generation. The entire folding process can generally be divided into several stages, which are then executed sequentially during the demonstration generation process.
• Figure 5: Keypoint-based recovery strategy. The figure illustrates an example of the recovery strategy. The recovery data are incorporated into the dataset and used jointly to train the end-to-end vision-action model. With these data, the model can learn how to retry when a grasp attempt fails.