A Dataset and Benchmark for Robotic Cloth Unfolding Grasp Selection: The ICRA 2024 Cloth Competition

TL;DR

This work tackles the lack of standardized benchmarks for robotic cloth manipulation by introducing the ICRA 2024 Cloth Competition, a rigorous, on-site benchmark focused on grasp pose selection for in-air unfolding and supported by a large public dataset. The dataset comprises 679 unfolding demonstrations across 34 garments (including a 503-trial training set with human-annotated grasps and 176 competition trials), enabling both learning-based and geometry-driven approaches to be evaluated under realistic conditions. Coverage is the primary objective metric, defined as $coverage = \frac{A_{current}}{A_{max}}$, with a $30$ second per-grasp execution cap to reflect real-time decision making; the evaluation also tracks grasp success and success-conditioned coverage. Results reveal that traditional geometric methods can outperform some learning-based approaches in this setting, while the dataset and live competition framework provide a valuable resource for improving data-driven cloth unfolding and for driving future benchmarks beyond the lab.

Abstract

Robotic cloth manipulation suffers from a lack of standardized benchmarks and shared datasets for evaluating and comparing different approaches. To address this, we created a benchmark and organized the ICRA 2024 Cloth Competition, a unique head-to-head evaluation focused on grasp pose selection for in-air robotic cloth unfolding. Eleven diverse teams participated in the competition, utilizing our publicly released dataset of real-world robotic cloth unfolding attempts and a variety of methods to design their unfolding approaches. Afterwards, we also expanded our dataset with 176 competition evaluation trials, resulting in a dataset of 679 unfolding demonstrations across 34 garments. Analysis of the competition results revealed insights about the trade-off between grasp success and coverage, the surprisingly strong achievements of hand-engineered methods and a significant discrepancy between competition performance and prior work, underscoring the importance of independent, out-of-the-lab evaluation in robotic cloth manipulation. The associated dataset is a valuable resource for developing and evaluating grasp selection methods, particularly for learning-based approaches. We hope that our benchmark, dataset and competition results can serve as a foundation for future benchmarks and drive further progress in data-driven robotic cloth manipulation. The dataset and benchmarking code are available at https://airo.ugent.be/cloth_competition.

Figures (9)

• Figure 1: Impression of the ICRA 2024 Cloth Competition on in-air unfolding. The dual-arm robotic setup, surrounded by participants and spectators, executes the final stretching motion to achieve a nearly perfectly unfolded T-shirt. All competition evaluations were conducted on-site at the conference in Yokohama, Japan, using this standardized, shared setup.
• Figure 2: Overview of the unfolding procedure. After the robot grasps the cloth at its highest and then its lowest point, the grasp selection algorithm must select a grasp pose based on the RGB-D observation. The robot then executes this grasp and stretches the cloth, unfolding it in the process. A final observation is recorded for evaluation.
• Figure 3: The hardware setup for the benchmark consists of a single RGB-D camera overlooking two UR5e robot arms. This image is from the AIRO lab setup, on which the training dataset was collected. Note that the wrist camera mounted on the right arm was not used for the competition.
• Figure 4: Examples from the competition of cloth unfolding results with varying coverage. Higher coverage generally corresponds to a more unfolded state, aligning with human assessment of task completion. Some grasps can be counterproductive, yielding lower coverage than not grasping at all.
• Figure 5: The color and depth images from the start and result observation from a dataset sample. The executed grasp is visualized on the start observation as an RGB coordinate frame representing the position and orientation of the gripper. The red axis indicates the direction along which the parallel gripper opens. By including the result observations, participants can estimate how effective the grasp was at unfolding.