Towards Safe Reinforcement Learning Using NMPC and Policy Gradients: Part II - Deterministic Case

TL;DR

This paper presents a methodology to deploy the stochastic policy gradient method, using actor-critic techniques, when the optimal policy is approximated using a parametric optimization problem, allowing one to enforce safety via hard constraints.

Abstract

In this paper, we present a methodology to deploy the deterministic policy gradient method, using actor-critic techniques, when the optimal policy is approximated using a parametric optimization problem, where safety is enforced via hard constraints. For continuous input space, imposing safety restrictions on the exploration needed to deploying the deterministic policy gradient method poses some technical difficulties, which we address here. We will investigate in particular policy approximations based on robust Nonlinear Model Predictive Control (NMPC), where safety can be treated explicitly. For the sake of brevity, we will detail the construction of the safe scheme in the robust linear MPC context only. The extension to the nonlinear case is possible but more complex. We will additionally present a technique to maintain the system safety throughout the learning process in the context of robust linear MPC. This paper has a companion paper treating the stochastic policy gradient case.

Figures (13)

• Figure 1: Illustration of the mean ($+$ symbol) and covariance (ellipsoids) \ref{['eq:MeanAndCov']} of the exploration \ref{['eq:ExploDefinition']} subject to safety constraints (solid red line). The deterministic policy is depicted as the $\times$ symbol, for the trivial problem \ref{['eq:TrivialExample']}, and different values of $\boldsymbol{\mathrm{\theta}}_{1,2}$ and $\boldsymbol{\mathrm{\theta}}_3=1$. Here the exploration is generated via \ref{['eq:NLP:Disturbed']}-\ref{['eq:d:distribution']} detailed below. One can observe that $\boldsymbol{\mathrm{\eta}}_{\boldsymbol{\mathrm{e}}}= 0$ is not possible to achieve when the policy is on the constraints, and that the covariances are impacted by the presence of the constraint. On the right-side graph, one can see how the covariance collapses when $\boldsymbol{\mathrm{\pi}}_{\boldsymbol{\mathrm{\theta}}}$ strongly activates the constraint.
• Figure 2: Illustration of the mean and covariance \ref{['eq:MeanAndCov']} of the exploration \ref{['eq:ExploDefinition']} subject to safety constraints (solid red line here), generated by the interior-point approach \ref{['eq:NLP:Disturbed:IP']} for two value of the relaxation parameter $\tau$. The policy is generated by \ref{['eq:TrivialExample']}, and the exploration by \ref{['eq:NLP:Disturbed']}-\ref{['eq:CostGradientDisturbance']}, with $\Sigma = I$. The mean estimator $\boldsymbol{\mathrm{c}}$ (+ symbol) and covariance estimator $M^{-1}$ (dashed ellipsoid) \ref{['eq:CovMean:Estimators']} are compared to the ones estimated by sampling (o symbol, solid line ellipsoid).
• Figure 3: Illustration of Proposition \ref{['eq:DetPolicy:WellDefined']} and the following discussion for the small problem \ref{['eq:TrivialExample']}. The solid black arrows represent the directions spanned by $\nabla_{\boldsymbol{\mathrm{\theta}}}\boldsymbol{\mathrm{\pi}}_{\boldsymbol{\mathrm{\theta}}}$ . The red, dashed-line arrows report the corresponding terms in $\frac{1}{\sigma}\mathbb{E}\left[\nabla_{\boldsymbol{\mathrm{\theta}}}\boldsymbol{\mathrm{\pi}}_{\boldsymbol{\mathrm{\theta}}}M\left(\boldsymbol{\mathrm{e}}-\boldsymbol{\mathrm{c}}\right)\boldsymbol{\mathrm{e}}^\top\right]$ appearing in \ref{['eq:PropositionTarget']}. The dotted-line blue arrows report the directions spanned by $\frac{\partial \boldsymbol{\mathrm{g}}}{\partial \boldsymbol{\mathrm{d}}}$. One can see that \ref{['eq:PropositionTarget']} holds for $\tau >0$ (left graph), and holds for parameters $\boldsymbol{\mathrm{\theta}}_{1,2}$ for $\tau\rightarrow 0$ (right graph) as they satisfy the assumptions of Proposition \ref{['eq:DetPolicy:WellDefined']}. However, \ref{['eq:PropositionTarget']} does not hold for $\boldsymbol{\mathrm{\theta}}_3$, as it influences the constraint \ref{['eq:TrivialExample:Const']}, and therefore violates the assumptions of Proposition \ref{['eq:DetPolicy:WellDefined']} (right graph). One can construe the problem as a lack of exploration (blue dotted-line arrows) in the direction $\nabla_{\boldsymbol{\mathrm{\theta}}_3}\boldsymbol{\mathrm{\pi}}_{\boldsymbol{\mathrm{\theta}}}$ due to the active constraint.
• Figure 4: Case 1. Evolution of the closed-loop performance $J$ over the RL steps. The solid line represents the estimation of $J$ based on the samples obtained in the batch. The dashed line represent the standard deviation due to the stochasticity of the system dynamics and policy disturbances.
• Figure 5: Case 1. Closed-loop system trajectories. The initial conditions $\boldsymbol{\mathrm{s}}_0$ are reported, as well as the target state reference ${\boldsymbol{\mathrm{x}}}_\mathrm{ref}$ (circle), and the MPC reference $\bar{\boldsymbol{\mathrm{x}}}$ at the first RL step and at the last one (grey and black $+$ symbol respectively). The trajectories at the first and last RL steps are reported as the light and dark grey polytopes. The solid black curve represents the state constraint $\|\boldsymbol{\mathrm{x}}\|^2 \leq 1$.

Theorems & Definitions (3)

• Proposition 1
• Proposition 2
• Proposition 3