OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World

TL;DR

Real-world reinforcement learning for robotics is hampered by safety concerns from unconstrained actions. The authors introduce OptLayer, a differentiable quadratic-programming layer that projects NN-predicted actions onto a safety-constrained feasible set, enabling safe execution during training and deployment. They show how to construct and solve the QP efficiently in batches and integrate it with TRPO-based policy optimization. Empirical results on 3D reaching with obstacle avoidance demonstrate improved safety and faster learning, including successful real-world robot experiments.

Abstract

While deep reinforcement learning techniques have recently produced considerable achievements on many decision-making problems, their use in robotics has largely been limited to simulated worlds or restricted motions, since unconstrained trial-and-error interactions in the real world can have undesirable consequences for the robot or its environment. To overcome such limitations, we propose a novel reinforcement learning architecture, OptLayer, that takes as inputs possibly unsafe actions predicted by a neural network and outputs the closest actions that satisfy chosen constraints. While learning control policies often requires carefully crafted rewards and penalties while exploring the range of possible actions, OptLayer ensures that only safe actions are actually executed and unsafe predictions are penalized during training. We demonstrate the effectiveness of our approach on robot reaching tasks, both simulated and in the real world.

Figures (8)

• Figure 1: 3D reaching with obstacle avoidance using reinforcement learning.
• Figure 2: 2D reacher description and rewards, with $d_\text{T}$ the distance between target and end-effector and $\beta_\text{coll} \in \{ 1, 5, 10, 50 \}$ a penalization coefficient.
• Figure 3: Reward and collision count for unconstrained 2D reaching task with collision penalization coefficient $\beta_\text{coll} \in \{1, 5, 10, 50\}$, averaged over three seeds. Moving average over $40$ consecutive episode rewards.
• Figure 4: OptLayer constrained optimization pipeline.
• Figure 5: Build a training trajectory for network $\mathcal{N}$ and optimization layer $\mathcal{O}$ in environment $\mathcal{E}$. If $constrained = \mathbf{false}$, execute raw predictions.