GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation

TL;DR

GarmentLab tackles the challenge of manipulating deformable garments by delivering a unified, GPU-based simulation and benchmark platform that combines multiple physics solvers (PBD and FEM), rich asset sets, and a first real-world deformable-object benchmark. The framework includes GarmentLab Engine (Omniverse Isaac Sim), GarmentLab Assets, and a 20-task GarmentLab Benchmark spanning five interaction categories, plus a Real-World Benchmark and a Sim2Real framework that integrates vision alignment and motion generation via ROS MoveIt and teleoperation. Through extensive simulations and real-world experiments, the authors show that current vision-based methods generalize better than RL policies in long-horizon garment tasks, while sim-to-real gaps can be mitigated with alignment, noise augmentation, and keypoint-based representations. The work provides a scalable, open-source path for developing and evaluating garment manipulation algorithms, facilitating transfer from simulation to real-world deployment. Overall, GarmentLab advances the state of the art by unifying simulation modalities, assets, benchmarks, and sim-to-real techniques tailored to deformable object manipulation.

Abstract

Manipulating garments and fabrics has long been a critical endeavor in the development of home-assistant robots. However, due to complex dynamics and topological structures, garment manipulations pose significant challenges. Recent successes in reinforcement learning and vision-based methods offer promising avenues for learning garment manipulation. Nevertheless, these approaches are severely constrained by current benchmarks, which offer limited diversity of tasks and unrealistic simulation behavior. Therefore, we present GarmentLab, a content-rich benchmark and realistic simulation designed for deformable object and garment manipulation. Our benchmark encompasses a diverse range of garment types, robotic systems and manipulators. The abundant tasks in the benchmark further explores of the interactions between garments, deformable objects, rigid bodies, fluids, and human body. Moreover, by incorporating multiple simulation methods such as FEM and PBD, along with our proposed sim-to-real algorithms and real-world benchmark, we aim to significantly narrow the sim-to-real gap. We evaluate state-of-the-art vision methods, reinforcement learning, and imitation learning approaches on these tasks, highlighting the challenges faced by current algorithms, notably their limited generalization capabilities. Our proposed open-source environments and comprehensive analysis show promising boost to future research in garment manipulation by unlocking the full potential of these methods. We guarantee that we will open-source our code as soon as possible. You can watch the videos in supplementary files to learn more about the details of our work. Our project page is available at: https://garmentlab.github.io/

Figures (11)

• Figure 1: GarmentLab provides realistic simulation for diverse garments with different physical propoerties, benchmarking various novel garment manipulation tasks in both simulation and the real world.
• Figure 2: The Architecture of GarmentLab.(Left) Built on PhysX5, our environment supports various simulation methods. (Middle) Our environment can deliver realistic simulations of diverse robots, garments, and interactions between multiple physics media. (Right) Subsequently, we can utilize these assets to construct tasks across various categories. (Bottom) The framework supports real-world deployment.
• Figure 3: GarmentLab Physics. GarmentLab explores the potential of different simulation methods, and provides different physical parameters, modeling the distinct properties of different materials in the real world.
• Figure 4: Diverse Tasks of GarmentLab Benchmark. We introduced 20 garment and deformable manipulation tasks including complicated long-horizon tasks. The last row shows the execution of these tasks in the real world.
• Figure 5: Real-World Benchmark. Part a demonstrates the whole pipeline of converting real-world objects into simulation assets. Part b demonstrates the performance of different categories of objects in both simulation and the real world (the first row), and the results of these objects being manipulated by the robot (the second row).