Contact-Safe Reinforcement Learning with ProMP Reparameterization and Energy Awareness

TL;DR

This paper addresses safe and robust control for contact-rich robotic manipulation by learning in task space with a ProMP-based trajectory prior, reinforced through PPO, while enforcing energy-aware, passivity-based safety using an energy-tank layer and Cartesian impedance control. The key idea is to refine smooth, compliant trajectories in weight space, condition trajectories via via-points, and gate execution to respect energy and power limits during contact. The authors demonstrate that this PPT framework yields higher task success, smoother trajectories, and safer interactions in both simulation (box pushing and maze sliding) and real-world experiments, with strong sim-to-real transfer and generalization to unseen contact conditions. The work offers a practical, safety-conscious pathway for deploying learning-based manipulation policies on real robots without task-specific retuning, potentially enabling more reliable deployment in cluttered, contact-rich environments.

Abstract

Reinforcement learning (RL) approaches based on Markov Decision Processes (MDPs) are predominantly applied in the robot joint space, often relying on limited task-specific information and partial awareness of the 3D environment. In contrast, episodic RL has demonstrated advantages over traditional MDP-based methods in terms of trajectory consistency, task awareness, and overall performance in complex robotic tasks. Moreover, traditional step-wise and episodic RL methods often neglect the contact-rich information inherent in task-space manipulation, especially considering the contact-safety and robustness. In this work, contact-rich manipulation tasks are tackled using a task-space, energy-safe framework, where reliable and safe task-space trajectories are generated through the combination of Proximal Policy Optimization (PPO) and movement primitives. Furthermore, an energy-aware Cartesian Impedance Controller objective is incorporated within the proposed framework to ensure safe interactions between the robot and the environment. Our experimental results demonstrate that the proposed framework outperforms existing methods in handling tasks on various types of surfaces in 3D environments, achieving high success rates as well as smooth trajectories and energy-safe interactions.

Figures (9)

• Figure 1: Safety-aware contact-rich environment. (A) Box pushing: The tool slides along the surface of the block to achieve translation and reorientation. Arrows indicate tangential motion and contact normals, while translucent snapshots illustrate temporal progression. (B) Maze sliding: The tool navigates unseen turns relying solely on contact feedback. Dashed lines represent the executed tool trajectory, and dots mark via-points used to condition the ProMP prior. A 5 cm scale bar is included for reference. This environment requires rich contact information regarding energy to ensure the safe manipulation of the box and navigation within the maze. Excessive energy may lead to damage to either the environment or the robot.
• Figure 2: Overview of the PPT framework. Observations and a phase variable are fed to the policy, which outputs residual ProMP weights. These are combined with via-point conditioned reference weights to form a ProMP trajectory. An energy-tank layer scales the command to ensure safe execution, which a Cartesian impedance controller tracks and converts to joint torques. Interaction feedback informs via-point reparameterization. PPO updates the policy using GAE and a value critic (dashed arrow).
• Figure 3: ProMP prior and via-point posterior (1D view). Left: trajectory represented as $y(\phi)=\boldsymbol{\Phi}(\phi)^\top \mathbf{w}$ with $K$ RBFs. Middle: prior $\mathrm{vec}(\mathbf{w})\sim\mathcal{N}(\boldsymbol{\mu}_w,\boldsymbol{\Sigma}_w)$: mean (red) $\pm 2\sigma$ band (blue). Right: posterior conditioned on via-points $\mathcal{D}$ (black $\times$), showing tightened uncertainty near constraints while preserving smoothness.
• Figure 4: End-effector tools for experiments.
• Figure 5: Training curves. Success rate (%) and max instantaneous power (W) vs. training steps for PP, PPT, S, ST. Higher success and lower power are better.