DexGarmentLab: Dexterous Garment Manipulation Environment with Generalizable Policy

TL;DR

DexGarmentLab tackles the challenge of dexterous garment manipulation by introducing a dedicated simulation environment with 15 tasks and 2500+ garment assets, built on Isaac Sim. It combines automated data collection from a single expert demonstration with HALO, a hierarchical policy that first identifies transferable manipulation affordances via GAM and then generates generalizable trajectories with SADP, a diffusion-based controller. The approach demonstrates data-efficiency and strong generalization in simulation and real-world tests, including two real-world deployment modes and sim-to-real transfer enhancements. By enabling scalable data collection and robust policy generalization across diverse garment geometries and deformations, DexGarmentLab pushes forward the development of autonomous, home-like dexterous garment manipulation. The work highlights practical implications for robot-assisted dressing, tailoring, and other everyday tasks involving deformable objects, while outlining realistic limitations and future improvements.

Abstract

Garment manipulation is a critical challenge due to the diversity in garment categories, geometries, and deformations. Despite this, humans can effortlessly handle garments, thanks to the dexterity of our hands. However, existing research in the field has struggled to replicate this level of dexterity, primarily hindered by the lack of realistic simulations of dexterous garment manipulation. Therefore, we propose DexGarmentLab, the first environment specifically designed for dexterous (especially bimanual) garment manipulation, which features large-scale high-quality 3D assets for 15 task scenarios, and refines simulation techniques tailored for garment modeling to reduce the sim-to-real gap. Previous data collection typically relies on teleoperation or training expert reinforcement learning (RL) policies, which are labor-intensive and inefficient. In this paper, we leverage garment structural correspondence to automatically generate a dataset with diverse trajectories using only a single expert demonstration, significantly reducing manual intervention. However, even extensive demonstrations cannot cover the infinite states of garments, which necessitates the exploration of new algorithms. To improve generalization across diverse garment shapes and deformations, we propose a Hierarchical gArment-manipuLation pOlicy (HALO). It first identifies transferable affordance points to accurately locate the manipulation area, then generates generalizable trajectories to complete the task. Through extensive experiments and detailed analysis of our method and baseline, we demonstrate that HALO consistently outperforms existing methods, successfully generalizing to previously unseen instances even with significant variations in shape and deformation where others fail. Our project page is available at: https://wayrise.github.io/DexGarmentLab/.

Figures (28)

• Figure 1: Overview. DexGarmentLab includes three major components: Environment, Automated Data Collection and Generalizable Policy. Firstly, we propose Dexterous Garment Manipulation Environment with 15 different task scenes (especially for bimanual coordination) based on 2500+ garments. Because of the same structure of category-level garment, category-level generalization is accessible, which empowers our proposed automated data collection pipeline to handle different position, deformation and shapes of garment with task config (including grasp position and task sequence) and grasp hand pose provided by single expert demonstration. With diverse collected demonstration data, we introduce Hierarchical gArment manipuLation pOlicy (HALO), combining affordance points and trajectories to generalize across different attributes in different tasks.
• Figure 2: Comparison of Garment-Robot Interactions between GarmentLab and DexGarmentLab.Left: In GarmentLab, Franka grasp garment with red block attached. Middle: We transfer AttachementBlock method to dexterours hands and set red block at the tip of each finger (ten blocks totally). The performance is not so good, as described in \ref{['PBD_design']}. Right: Our method (DexGarmentLab) can make the interactions between dexterous hands and garment more natural.
• Figure 3: Compare Garment Properties between GarmentLab and DexGarmentLab.Left: After folding in GarmentLab, the garment struggles to maintain a stable folded state and easily becomes disorganized. Right: With realistic physical simulation in DexGarmentLab, folded garments can stably maintain their folded states, which means Garment Properties is more mature and natural.
• Figure 4: DexGarmentLab Simulation Environment. We introduce 15 garment manipulation tasks across 8 categories, encompassing both garment-self-interaction (e.g., Fling, Fold) and garment-environment-interaction (e.g., Hang, Wear, Store) scenarios. In garment-self-interaction tasks, key variables include garment position, orientation, and shape. In garment-environment-interaction tasks, environment-interaction assets positions (e.g., hangers, pothooks, humans, etc.) are also considered.
• Figure 5: Automated Data Collection Pipeline. Given a single expert demonstration, we can get demo points, demo task sequences and demo grasp poses for the specific task. Category-level garment (w/ or w/o deformation) has almost the same structure, base on which we can train Garment Affordance Model (GAM) with category-level generalization. With GAM (refer Sec. \ref{['data:model']}), we match demo points from the demo garment point cloud $O$ to a new garment point cloud $O'$ and control robot to execute the specific task based on the demo task sequences (through trajectory retargeting) with dexhands' movement guided by demo hand grasp poses (through PD controller based on joint positions). 'Fold Tops' task is shown as example in this figure.