SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation

TL;DR

SoftGym presents a standardized, open-source benchmark suite for deformable object manipulation built on the FleX simulator, enabling reproducible evaluation of RL approaches across rope, cloth, and fluid tasks. By examining full-state, reduced-state, and image-based observation modalities, the study reveals that image-based methods generally underperform compared to state-informed baselines, while high-dimensional dynamics pose a persistent learning challenge. The work introduces Dynamics and State Oracles as upper-bound and informed baselines and demonstrates sim2real potential via initial reality-gap experiments, highlighting directions for future algorithm development. Overall, SoftGym offers a rigorous platform for fair comparison and rapid progress in deformable-object manipulation research.

Abstract

Manipulating deformable objects has long been a challenge in robotics due to its high dimensional state representation and complex dynamics. Recent success in deep reinforcement learning provides a promising direction for learning to manipulate deformable objects with data driven methods. However, existing reinforcement learning benchmarks only cover tasks with direct state observability and simple low-dimensional dynamics or with relatively simple image-based environments, such as those with rigid objects. In this paper, we present SoftGym, a set of open-source simulated benchmarks for manipulating deformable objects, with a standard OpenAI Gym API and a Python interface for creating new environments. Our benchmark will enable reproducible research in this important area. Further, we evaluate a variety of algorithms on these tasks and highlight challenges for reinforcement learning algorithms, including dealing with a state representation that has a high intrinsic dimensionality and is partially observable. The experiments and analysis indicate the strengths and limitations of existing methods in the context of deformable object manipulation that can help point the way forward for future methods development. Code and videos of the learned policies can be found on our project website.

Figures (6)

• Figure 1: Visualizations of all tasks in SoftGym. These tasks can be used to evaluate how well an algorithm works on a variety of deformable object manipulation tasks.
• Figure 2: Normalized performance at the last time step of the episode of all the algorithms on the evaluation set. The x-axis shows the number of training time steps.
• Figure 3: Bottom row: Open-loop prediction of PlaNet. Given an initial set of five frames, PlaNet predicts the following 30 frames. Here we show the last observed frame in the first column and four evenly spaced key frames out of the 30 predicted frames in the last four columns. Top row: Ground-truth future observations.
• Figure 4: Two pick-and-place rollouts both in simulation and in the real world for a cloth manipulation task. For each rollout, the left column shows the simulation; the right shows the real world.
• Figure 5: Illustration of task variations. Each image shows the task after the initial reset. Variations of tasks in SoftGym-Hard are omitted here due to similarity to the ones shown.