Towards Safe Reinforcement Learning Using NMPC and Policy Gradients: Part I - Stochastic case

TL;DR

This work addresses safe reinforcement learning in continuous-control settings by embedding the policy within constrained parametric NLPs, specifically robust NMPC, to enforce hard safety constraints. It develops an optimization-based approach for constructing safe stochastic policies and derives tractable gradients for policy learning using interior-point based sensitivity analysis, focusing on robust linear MPC as a practical instantiation. The key contributions include a computationally efficient gradient computation for the stochastic policy, a data-driven method to preserve safety during learning, and an illustrative robust linear MPC example demonstrating improved closed-loop performance while maintaining safety. The work advances safe RL by bridging stochastic policy gradients with optimization-based safety guarantees, with practical implications for real-time control under uncertainty and disturbances.

Abstract

We present a methodology to deploy the stochastic policy gradient method, using actor-critic techniques, when the optimal policy is approximated using a parametric optimization problem, allowing one to enforce safety via hard constraints. For continuous input spaces, imposing safety restrictions on the stochastic policy can make the sampling and evaluation of its density difficult. This paper proposes a computationally effective approach to solve that issue. We will focus on policy approximations based on robust Nonlinear Model Predictive Control (NMPC), where safety can be treated explicitly. For the sake of brevity, we will detail safe policies in the robust linear MPC context only. The extension to the nonlinear case is possible but more complex. We will additionally present a technique to maintain the system safety throughout the learning process in the context of robust linear MPC. This paper has a companion paper treating the deterministic policy gradient case.

Figures (12)

• Figure 1: Illustration of the stochastic policy resulting from \ref{['eq:RobustNMPC:Disturbed:Policy']}-\ref{['eq:dtoa']} for different values of $\tau$ for a fixed $\boldsymbol{\mathrm{s}}$, and $\boldsymbol{\mathrm{u}}_0^{\mathrm d}$ restricted within a set $\mathbb{S}(\boldsymbol{\mathrm{s}})$ depicted as the solid line. The resulting probability density $\pi^\tau_{\boldsymbol{\mathrm{\theta}}}\left[\boldsymbol{\mathrm{a}}|\boldsymbol{\mathrm{s}}\right]$ is constrained to remain within $\mathbb{S}(\boldsymbol{\mathrm{s}})$. For very low values of $\tau$, the density tends to a Dirac-like distribution on the border of the set, see right-side graph.
• Figure 2: Illustration of the gradient of the stochastic policy resulting from \ref{['eq:RobustNMPC:Disturbed:Policy']}-\ref{['eq:dtoa']} for different values of $\tau$, $\boldsymbol{\mathrm{s}}$ fixed, and $\boldsymbol{\mathrm{u}}_0^{\mathrm d}$ restricted within a set $\mathbb{S}(\boldsymbol{\mathrm{s}})$ depicted as the solid circle. The first row of graphs depict the gradient with respect to parameter $\boldsymbol{\mathrm{\theta}}_1$ for which $\nabla_{\boldsymbol{\mathrm{\theta}}_1}\boldsymbol{\mathrm{h}},\,\nabla_{\boldsymbol{\mathrm{\theta}}_1}\boldsymbol{\mathrm{f}}=0$, while the second row depicts the gradient with respect to parameter $\boldsymbol{\mathrm{\theta}}_2$ for which $\nabla_{\boldsymbol{\mathrm{\theta}}_2}\boldsymbol{\mathrm{h}}\neq0$. The last row depicts the conditioning of matrix $\frac{\partial\boldsymbol{\mathrm{g}}}{\partial \boldsymbol{\mathrm{d}}}$. As predicted by Proposition \ref{['eq:StochPolicy:WellDefined']} for $\tau\rightarrow 0$, $\frac{\partial\boldsymbol{\mathrm{g}}}{\partial \boldsymbol{\mathrm{d}}}$ tends to a rank-deficient matrix for $\boldsymbol{\mathrm{a}} = \boldsymbol{\mathrm{u}}_0^{\mathrm d}\rightarrow \partial\mathbb{S}\left(\boldsymbol{\mathrm{s}}\right)$, and the gradient $\nabla_{\boldsymbol{\mathrm{\theta}}_2}\log \boldsymbol{\mathrm{\pi}}_{\boldsymbol{\mathrm{\theta}}}$ degenerates while $\nabla_{\boldsymbol{\mathrm{\theta}}_1}\log \boldsymbol{\mathrm{\pi}}_{\boldsymbol{\mathrm{\theta}}}$ does not.
• Figure 3: Case 1. Evolution of the closed-loop performance $J$ over the RL steps. The solid line represents the estimation of $J$ based on the samples obtained in the batch. The dashed line represent the standard deviation due to the stochasticity of the system dynamics and policy disturbances.
• Figure 4: Case 1. Closed-loop system trajectories. The initial conditions $\boldsymbol{\mathrm{s}}_0$ are reported, as well as the target state reference ${\boldsymbol{\mathrm{x}}}_\mathrm{ref}$ (circle), and the MPC reference $\bar{\boldsymbol{\mathrm{x}}}$ at the first RL step and at the last one (grey and black $+$ symbol respectively). The trajectories at the first and last RL steps are reported as the light and dark grey polytopes. The solid black curve represents the state constraint $\|\boldsymbol{\mathrm{x}}\|^2 \leq 1$.
• Figure 5: Case 1. Evolution of the nominal MPC model over the RL steps. We report here the difference between the nominal model used in the MPC scheme and the real system.

Theorems & Definitions (4)

• Lemma 1
• Lemma 2
• Proposition 1
• Proposition 2