Self-Curriculum Model-based Reinforcement Learning for Shape Control of Deformable Linear Objects

TL;DR

A two-stage framework combining Reinforcement Learning (RL) and online visual servoing that enables efficient policy learning and significantly outperforms mainstream baselines in shape control success rate and precision is proposed.

Abstract

Precise shape control of Deformable Linear Objects (DLOs) is crucial in robotic applications such as industrial and medical fields. However, existing methods face challenges in handling complex large deformation tasks, especially those involving opposite curvatures, and lack efficiency and precision. To address this, we propose a two-stage framework combining Reinforcement Learning (RL) and online visual servoing. In the large-deformation stage, a model-based reinforcement learning approach using an ensemble of dynamics models is introduced to significantly improve sample efficiency. Additionally, we design a self-curriculum goal generation mechanism that dynamically selects intermediate-difficulty goals with high diversity through imagined evaluations, thereby optimizing the policy learning process. In the small-deformation stage, a Jacobian-based visual servo controller is deployed to ensure high-precision convergence. Simulation results show that the proposed method enables efficient policy learning and significantly outperforms mainstream baselines in shape control success rate and precision. Furthermore, the framework effectively transfers the policy trained in simulation to real-world tasks with zero-shot adaptation. It successfully completes all 30 cases with diverse initial and target shapes across DLOs of different sizes and materials. The project website is available at: https://anonymous.4open.science/w/sc-mbrl-dlo-EB48/

Figures (9)

• Figure 1: Illustration of the task with large deformations exhibiting opposite curvatures. The initial and target shapes have completely opposite concavities. The policy is trained in a simulator and transferred to the real-world scenario.
• Figure 2: Overview of the proposed DLO shape control framework. In the large-deformation stage, policy is trained with transition augmentation using ensemble model, while a self-curriculum module adaptively selects interaction goals. In the small-deformation stage, an online visual servo controller refines the motion to achieve precise convergence.
• Figure 3: The architecture of the MBRL framework with ensemble dynamics models.
• Figure 4: The proposed self-curriculum goal generation method. (a) Candidate goals are randomly sampled from the replay buffer. (b) Imagined evaluations are conducted to categorize intermediate-difficulty goals. (c) Weighted Farthest Point Sampling (FPS) selects the final interaction goal set $\mathcal{G}_{\text{interact}}$.
• Figure 5: Policy learning curves under two initial conditions: (a) straight-line initialization and (b) diverse-shape initialization.