One Fling to Goal: Environment-aware Dynamics for Goal-conditioned Fabric Flinging

TL;DR

This work tackles dynamic, goal-conditioned fabric manipulation in diverse environments, aiming to place a fabric into a predefined state $S_g$ from an initial state $S_t$ in a single fling. It introduces an environment-aware graph-based dynamics model that predicts node velocities and accounts for interactions with surroundings via environmental descriptors, coupled with a real-time model predictive control policy to refine the fling trajectory. The key contributions include (i) a single-shot goal-conditioned manipulation framework, (ii) a dynamics model that integrates environmental cues, and (iii) a two-stage control strategy that enables accurate trajectory refinement during execution. Validated across five simulated scenarios and real-world zero-shot experiments, the approach demonstrates robust generalization to unseen fabrics and environments, highlighting its potential for practical robotic fabric handling.

Abstract

Fabric manipulation dynamically is commonly seen in manufacturing and domestic settings. While dynamically manipulating a fabric piece to reach a target state is highly efficient, this task presents considerable challenges due to the varying properties of different fabrics, complex dynamics when interacting with environments, and meeting required goal conditions. To address these challenges, we present \textit{One Fling to Goal}, an algorithm capable of handling fabric pieces with diverse shapes and physical properties across various scenarios. Our method learns a graph-based dynamics model equipped with environmental awareness. With this dynamics model, we devise a real-time controller to enable high-speed fabric manipulation in one attempt, requiring less than 3 seconds to finish the goal-conditioned task. We experimentally validate our method on a goal-conditioned manipulation task in five diverse scenarios. Our method significantly improves this goal-conditioned task, achieving an average error of 13.2mm in complex scenarios. Our method can be seamlessly transferred to real-world robotic systems and generalized to unseen scenarios in a zero-shot manner.

Figures (8)

• Figure 1: Goal-conditioned dynamic manipulation task in complex environments. (Left) Fabric manipulation in manufacturing and domestic settings. (Right) Experiment setup. We extract the depth information from an RGBD camera and adjust the picker movement in real-time to achieve the final target state. We also validate this policy across complex environments.
• Figure 2: Overview of our proposed algorithm. The first stage (top) involves sampling a batch of picker trajectories based on the current and target states before executing the action. The dynamics model predicts the cloth state after execution, and the distance to the target state is evaluated using Eq. \ref{['eq:distance']}. The best trajectory is selected. In the second stage (bottom), fine adjustments are made at each action based on the current visual observation, allowing for either more forceful or gentler movements.
• Figure 3: Reduced action space parameterized by the turning point. (A) In the first stage, a batch of trajectories parameterized by a turning point are sampled. (B) In the second stage, we refine the trajectory by adding delta actions and select the best one based on real-time visual feedback and a trained, dynamic model.
• Figure 4: Dynamics model with environmental awareness. A) The state of the fabric is represented as a graph representation augmented with environmental information and is fed into the dynamics model. The velocity for each point in the graph is predicted. B) The dynamics model can predict the velocity of each point well by augmenting the fabric state with environmental information. Points in contact with the rigid objects in the environment will slide on the surface of the rigid object.
• Figure 5: Qualitative results of the real-world goal-conditioned manipulation experiments. The first four rows showcase four rigid objects similar to those in the simulation, including an elevated platform, a hemisphere, a wireframe mimicking a pole, and a stool mimicking a table. The fifth row demonstrates our method's ability to generalize to an unseen fabric piece with a concave shape. In the final row, our method is applied to fold a fabric piece in half, in addition to previous unfolding or hanging tasks.