Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Gradient-based Model Free Optimal LQG Control with a Probabilistic Risk Constraint

Arunava Naha, Subhrakanti Dey

TL;DR

This paper investigates a model-free optimal control design that minimizes an infinite horizon average expected quadratic cost of states and control actions subject to a probabilistic risk or chance constraint using input-output data and designs an optimal controller within the class of linear state feedback control.

Abstract

In this paper, we investigate a model-free optimal control design that minimizes an infinite horizon average expected quadratic cost of states and control actions subject to a probabilistic risk or chance constraint using input-output data. In particular, we consider linear time-invariant systems and design an optimal controller within the class of linear state feedback control. Three different policy gradient (PG) based algorithms, natural policy gradient (NPG), Gauss-Newton policy gradient (GNPG), and deep deterministic policy gradient (DDPG), are developed, and compared with the optimal risk-neutral linear-quadratic regulator (LQR) and a scenario-based model predictive control (MPC) technique via numerical simulations. The convergence properties and the accuracy of all the algorithms are compared numerically. We also establish analytical convergence properties of the NPG and GNPG algorithms under the known model scenario, while the proof of convergence for the unknown model scenario is part of our ongoing work.

Policy Gradient-based Model Free Optimal LQG Control with a Probabilistic Risk Constraint

TL;DR

This paper investigates a model-free optimal control design that minimizes an infinite horizon average expected quadratic cost of states and control actions subject to a probabilistic risk or chance constraint using input-output data and designs an optimal controller within the class of linear state feedback control.

Abstract

In this paper, we investigate a model-free optimal control design that minimizes an infinite horizon average expected quadratic cost of states and control actions subject to a probabilistic risk or chance constraint using input-output data. In particular, we consider linear time-invariant systems and design an optimal controller within the class of linear state feedback control. Three different policy gradient (PG) based algorithms, natural policy gradient (NPG), Gauss-Newton policy gradient (GNPG), and deep deterministic policy gradient (DDPG), are developed, and compared with the optimal risk-neutral linear-quadratic regulator (LQR) and a scenario-based model predictive control (MPC) technique via numerical simulations. The convergence properties and the accuracy of all the algorithms are compared numerically. We also establish analytical convergence properties of the NPG and GNPG algorithms under the known model scenario, while the proof of convergence for the unknown model scenario is part of our ongoing work.
Paper Structure (19 sections, 6 theorems, 40 equations, 6 figures, 1 table, 5 algorithms)

This paper contains 19 sections, 6 theorems, 40 equations, 6 figures, 1 table, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Approach and Contributions
  4. Organization
  5. Notations
  6. Problem Formulation
  7. Reward Structure
  8. PG-based AC Algorithms under study
  9. Natural Policy Gradient (NPG) based AC algorithm
  10. Gauss-Newton Policy Gradient (GNPG)
  11. Deep Deterministic Policy Gradient (DDPG)
  12. Analytical Results
  13. Finding the optimal value of the Lagrange multiplier $\lambda$
  14. Numerical Results
  15. Conclusion
  16. ...and 4 more sections

Key Result

Lemma 1

For a fixed $\lambda > 0$, the Lagrangian function $\mathcal{L}(\bf K, \lambda)$ given by (eqn:LK) is coercive on $\mathcal{K}$ in the sense that $\mathcal{L}(\bf K, \lambda) \rightarrow \infty$ as $\bf K \rightarrow \delta \mathcal{K}$, where $\delta \mathcal{K}$ denotes the boundary of $\mathcal{K

Figures (6)

  • Figure 1: Average return $\mathcal{R}$. $\lambda = 100$, $J_c = 8.7\%$ (NPG).
  • Figure 2: Constraint violation probability $J_c$ and control cost $J$ for different $\lambda$ values.
  • Figure 3: Control cost $J$ vs. Constraint violation probability $J_c$. $\lambda = [1,5,10,15,20,50,100]$.
  • Figure 4: Norm of the policy gradient. $\lambda = 10$, $J_c = 16.5\%$ (NPG).
  • Figure 5: Critic loss. $\lambda = 10$, $J_c = 16.5\%$ (NPG).
  • ...and 1 more figures

Theorems & Definitions (20)

  • Remark 1
  • Remark 2
  • Lemma 1
  • proof
  • Remark 3
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Remark 4
  • ...and 10 more