RTFF: Random-to-Target Fabric Flattening Policy using Dual-Arm Manipulator

TL;DR

This paper tackles the problem of flattening and aligning randomly wrinkled fabrics to arbitrary wrinkle-free targets in garment processes. It introduces RTFF, a hybrid imitation-learning–visual servoing policy that uses a template-based fabric mesh to provide a correspondence-preserving state representation and seamless IL–VS switching, enabling robust manipulation under occlusions. Central to the approach is MACT, a Transformer-based IL policy conditioned on mesh inputs and alignment error, which, together with a dual-arm VS controller and impedance safety, achieves accurate, generalizable fabric flattening demonstrated on a real dual-arm system. The framework demonstrates strong zero-shot generalization across fabrics and scales, reducing data requirements and improving manipulation reliability in practical textile applications.

Abstract

Robotic fabric manipulation in garment production for sewing, cutting, and ironing requires reliable flattening and alignment, yet remains challenging due to fabric deformability, effectively infinite degrees of freedom, and frequent occlusions from wrinkles, folds, and the manipulator's End-Effector (EE) and arm. To address these issues, this paper proposes the first Random-to-Target Fabric Flattening (RTFF) policy, which aligns a random wrinkled fabric state to an arbitrary wrinkle-free target state. The proposed policy adopts a hybrid Imitation Learning-Visual Servoing (IL-VS) framework, where IL learns with explicit fabric models for coarse alignment of the wrinkled fabric toward a wrinkle-free state near the target, and VS ensures fine alignment to the target. Central to this framework is a template-based mesh that offers precise target state representation, wrinkle-aware geometry prediction, and consistent vertex correspondence across RTFF manipulation steps, enabling robust manipulation and seamless IL-VS switching. Leveraging the power of mesh, a novel IL solution for RTFF-Mesh Action Chunking Transformer (MACT)-is then proposed by conditioning the mesh information into a Transformer-based policy. The RTFF policy is validated on a real dual-arm tele-operation system, showing zero-shot alignment to different targets, high accuracy, and strong generalization across fabrics and scales. Project website: https://kaitang98.github.io/RTFF_Policy/

Figures (6)

• Figure 1: RTFF task.
• Figure 2: Overview of the proposed RTFF policy framework.
• Figure 3: MACT policy architecture.
• Figure 4: Simulated and real fabric mesh prediction.
• Figure 5: Example RTFF sequences with (top) and without (bottom) Visual Servoing (VS).

Theorems & Definitions (2)

• Definition 1: RTFF task
• Definition 2: RTFF policy