Actor-Critic Learning for Risk-Constrained Linear Quadratic Regulation
Weijian Li, Andreas A. Malikopoulos
TL;DR
This work tackles the risk-constrained infinite-horizon LQR problem with unknown system dynamics by formulating it as a constrained, model-free optimization and solving it via a primal–dual, multi-time-scale actor–critic algorithm. The critic uses temporal-difference learning to estimate the action-value function, while the actor performs policy gradient updates on an affine policy and the dual variable enforces the risk constraint through gradient ascent on the Lagrangian $\mathcal{L}(X,\mu)$. The authors establish almost-sure convergence of all iterates under standard stochastic-approximation conditions and demonstrate the method on a simulated LTI system, showing that the learned policy approaches the optimal affine policy $u_t = -K^* x_t + b^*$ while satisfying the risk bound $\mathcal{J}_c(X) \le \bar{\iota}$. The approach advances safe data-driven control by providing online, model-free guarantees and a scalable, sample-efficient learning framework for risk-aware regulation.
Abstract
In this paper, we investigate the infinite-horizon risk-constrained linear quadratic regulator problem (RC-QR), which augments the classical LQR formulation with a statistical constraint on the variability of the system state to incorporate risk awareness, a key requirement in safety-critical control applications. We propose an actor-critic learning algorithm that jointly performs policy evaluation and policy improvement in a model-free and online manner. The RC-QR problem is first reformulated as a max-min optimization problem, from which we develop a multi-time-scale stochastic approximation scheme. The critic employs temporal-difference learning to estimate the action-value function, the actor updates the policy parameters via a policy gradient step, and the dual variable is adapted through gradient ascent to enforce the risk constraint.