Risk-Aware Safe Reinforcement Learning for Control of Stochastic Linear Systems
Babak Esmaeili, Nariman Niknejad, Hamidreza Modares
TL;DR
This work addresses safe reinforcement learning for stochastic discrete-time LTI systems by decoupling safety from performance and learning a risk-aware safe backup controller alongside the RL policy. It constructs a large, probabilistically safe region inside a polyhedral admissible set via the convex hull of ellipsoids, employing both model-based and data-driven (certainty-equivalence and minimum-variance) approaches to certify probabilistic safety with minimal data. A low-dimensional interpolation between the RL policy and the learned safe backup preserves RL performance while enforcing safety under noise, and set-partitioning yields scalable, piecewise-affine safeguards. Simulation and a car lane-keeping example demonstrate that the proposed variance-aware safety design reduces safety-violation risk without severely compromising performance, offering a practical framework for safe RL in uncertain, data-limited environments.
Abstract
This paper presents a risk-aware safe reinforcement learning (RL) control design for stochastic discrete-time linear systems. Rather than using a safety certifier to myopically intervene with the RL controller, a risk-informed safe controller is also learned besides the RL controller, and the RL and safe controllers are combined together. Several advantages come along with this approach: 1) High-confidence safety can be certified without relying on a high-fidelity system model and using limited data available, 2) Myopic interventions and convergence to an undesired equilibrium can be avoided by deciding on the contribution of two stabilizing controllers, and 3) highly efficient and computationally tractable solutions can be provided by optimizing over a scalar decision variable and linear programming polyhedral sets. To learn safe controllers with a large invariant set, piecewise affine controllers are learned instead of linear controllers. To this end, the closed-loop system is first represented using collected data, a decision variable, and noise. The effect of the decision variable on the variance of the safe violation of the closed-loop system is formalized. The decision variable is then designed such that the probability of safety violation for the learned closed-loop system is minimized. It is shown that this control-oriented approach reduces the data requirements and can also reduce the variance of safety violations. Finally, to integrate the safe and RL controllers, a new data-driven interpolation technique is introduced. This method aims to maintain the RL agent's optimal implementation while ensuring its safety within environments characterized by noise. The study concludes with a simulation example that serves to validate the theoretical results.