Automated Action Generation based on Action Field for Robotic Garment Manipulation

TL;DR

This work tackles the challenge of robotic garment manipulation by introducing a dense, pixel-wise action generator that outputs a 3-channel action field (score, angle, distance) from a single RGB image. Actions are represented as a planar 4D vector $[x, y, heta, d]$ and realized through a ResNet-18-based encoder–decoder that yields continuous, dense predictions in one forward pass, improving both accuracy and speed over discretized, multi-pass approaches. The training framework combines a tailored score loss, angle/distance regression losses, and a novel shape loss that aligns the garment toward a target mask, with extensive simulator data and real-world validation showing strong generalization and sim-to-real transfer. These contributions enable faster, more robust garment unfolding and alignment across diverse garment types, with practical impact for automation in textile manufacturing and garment handling.

Abstract

Garment manipulation using robotic systems is a challenging task due to the diverse shapes and deformable nature of fabric. In this paper, we propose a novel method for robotic garment manipulation that significantly improves the accuracy while reducing computational time compared to previous approaches. Our method features an action generator that directly interprets scene images and generates pixel-wise end-effector action vectors using a neural network. The network also predicts a manipulation score map that ranks potential actions, allowing the system to select the most effective action. Extensive simulation experiments demonstrate that our method achieves higher unfolding and alignment performances and faster computation time than previous approaches. Real-world experiments show that the proposed method generalizes well to different garment types and successfully flattens garments.

Figures (16)

• Figure 1: (a) is the robot platform and (b) shows the end-effector used to manipulate the fabric.
• Figure 2: (a) The action that manipulates the fabric contains the start point $(x, y)$ in the image, the moving distance $d$, and the moving angle $\theta$. (b) demonstrates the sliding effect of the fabric by applying the action.
• Figure 3: Framework of the proposed dense action generator.
• Figure 4: (a) shows the garment mask, and (b) shows the enlarged action areas that are filtered by the garment mask. (c) is the input image and (d) is the generated score map without action enlargement, which incorrectly focuses on the background and can not differentiate the areas in the garment or fabric.
• Figure 5: The state images and the action field involved in calculating the shape loss. (a) shows the mask of the garment in its current state. (b) is the action field determined by the predicted angle and distance map. (c) is the result of applying the dense actions (b) to each pixel in the mask of (a). (d) is the mask representing the target state of the garment in the alignment task.