Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Almost Surely $\sqrt{T}$ Regret for Adaptive LQR

Yiwen Lu, Yilin Mo

TL;DR

This work addresses adaptive LQR with unknown system matrices $A,B$ under Gaussian noise by proposing a certainty-equivalent controller augmented with a circuit-breaking mechanism and persistent excitation via probing noise. It proves an almost-sure regret bound of $\\tilde{O}(\\sqrt{T})$, made possible by carefully bounding random times $T_{ ext{stab}}$ and $T_{ ext{nocb}}$, and by showing circuit-breaking is triggered only finitely often, thus not affecting asymptotic performance. The theory accommodates open-loop unstable systems through a pre-stabilization framework and is validated through simulations on a Tennessee Eastman Process–like plant and a cart-pole example, demonstrating robust convergence and safety guarantees. The results advance the state of adaptive LQR by moving from probabilistic guarantees to almost-sure performance with explicit time-to-stabilization and switching behavior, with potential impact on safety-critical control applications.

Abstract

The Linear-Quadratic Regulation (LQR) problem with unknown system parameters has been widely studied, but it has remained unclear whether $\tilde{ \mathcal{O}}(\sqrt{T})$ regret, which is the best known dependence on time, can be achieved almost surely. In this paper, we propose an adaptive LQR controller with almost surely $\tilde{ \mathcal{O}}(\sqrt{T})$ regret upper bound. The controller features a circuit-breaking mechanism, which circumvents potential safety breach and guarantees the convergence of the system parameter estimate, but is shown to be triggered only finitely often and hence has negligible effect on the asymptotic performance of the controller. The proposed controller is also validated via simulation on Tennessee Eastman Process~(TEP), a commonly used industrial process example.

Almost Surely $\sqrt{T}$ Regret for Adaptive LQR

TL;DR

This work addresses adaptive LQR with unknown system matrices under Gaussian noise by proposing a certainty-equivalent controller augmented with a circuit-breaking mechanism and persistent excitation via probing noise. It proves an almost-sure regret bound of , made possible by carefully bounding random times and , and by showing circuit-breaking is triggered only finitely often, thus not affecting asymptotic performance. The theory accommodates open-loop unstable systems through a pre-stabilization framework and is validated through simulations on a Tennessee Eastman Process–like plant and a cart-pole example, demonstrating robust convergence and safety guarantees. The results advance the state of adaptive LQR by moving from probabilistic guarantees to almost-sure performance with explicit time-to-stabilization and switching behavior, with potential impact on safety-critical control applications.

Abstract

The Linear-Quadratic Regulation (LQR) problem with unknown system parameters has been widely studied, but it has remained unclear whether regret, which is the best known dependence on time, can be achieved almost surely. In this paper, we propose an adaptive LQR controller with almost surely regret upper bound. The controller features a circuit-breaking mechanism, which circumvents potential safety breach and guarantees the convergence of the system parameter estimate, but is shown to be triggered only finitely often and hence has negligible effect on the asymptotic performance of the controller. The proposed controller is also validated via simulation on Tennessee Eastman Process~(TEP), a commonly used industrial process example.
Paper Structure (27 sections, 17 theorems, 98 equations, 5 figures, 1 table, 1 algorithm)

This paper contains 27 sections, 17 theorems, 98 equations, 5 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Controller design
  4. Theoretical results
  5. Main results
  6. Extension to open-loop unstable systems
  7. Simulation
  8. Result on an open-loop stable system
  9. Result on an open-loop unstable system
  10. Conclusion
  11. Proof of Theorem \ref{['thm:prop']}
  12. Proof of Theorem \ref{['thm:prop']}, item \ref{['item:Enoise']})
  13. Proof of Theorem \ref{['thm:prop']}, item \ref{['item:Ecov']})
  14. Proof of Theorem \ref{['thm:prop']}, item \ref{['item:xnorm']})
  15. Proof of Theorem \ref{['thm:prop']}, item \ref{['item:Ecross']})
  16. ...and 12 more sections

Key Result

Theorem 4

Let $n,m$ be the dimensions of the state and input vectors respectively, and $P_0, \rho_0, P^*, \rho^*$ be defined in eq:P0, eq:rho0, eq:dare, eq:rho_star respectively. Let $\bar{n} = m + n$, then the following properties hold: Here, the notation "$\lesssim$" means that the left-hand side is bounded by the right-hand side up to a constant factor, where the constant depends on the system parameter

Figures (5)

  • Figure 1: Block diagram of the closed-loop system under the proposed controller. The control input is the superimposition of a deterministic input $u^{cb}$ and a random probing input $u^{pr}$. The deterministic part $u^{cb}$ is normally the same as the certainty equivalent input $u^{ce}$, but takes the value zero when circuit-breaking is triggered, where $\xi$ is a counter for circuit-breaking. The certainty equivalent gain is updated using the parameter estimator in the meantime.
  • Figure 2: Double-log plot of average regret against time step
  • Figure 3: Regret divided by $\sqrt{T}$ against time step
  • Figure 4: Histogram of $T_{\text{nocb}}$ among all sample paths
  • Figure 5: Double-log plot of average regret against time step in open-loop unstable cart-pole system. Five random sample paths are shown.

Theorems & Definitions (46)

  • Remark 1
  • Remark 2
  • Remark 3
  • Theorem 4: Properties of closed-loop system
  • proof : Proof Sketch
  • Remark 5
  • Remark 6
  • Corollary 7
  • proof
  • Theorem 8: Non-asymptotic regret bound
  • ...and 36 more