Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulator

Xuyang Chen, Jingliang Duan, Lin Zhao

TL;DR

This paper proves that a classic single-sample single-timescale actor-critic algorithm can achieve a global $\epsilon$-optimal policy for the continuous-state, continuous-action LQR problem, with a sample complexity of $\mathcal{O}(\epsilon^{-2})$. It develops a finite-time analysis by viewing the three interconnected updates (cost estimation, critic TD, and actor natural-gradient) as an interconnected system and establishing convergence of the cost-estimator error, critic error, and gradient error at $\mathcal{O}(T^{-1/2})$ under a suitable step-size ratio $c$. A gradient-domination property for LQR links the gradient norm to policy performance, enabling global convergence guarantees, while experiments validate improved sample efficiency over baselines. The results bridge theory and practice for AC in continuous control, highlighting how rough gradient estimates can still yield globally optimal policies with proper coupling analysis and stability constraints.

Abstract

Actor-critic methods have achieved state-of-the-art performance in various challenging tasks. However, theoretical understandings of their performance remain elusive and challenging. Existing studies mostly focus on practically uncommon variants such as double-loop or two-timescale stepsize actor-critic algorithms for simplicity. These results certify local convergence on finite state- or action-space only. We push the boundary to investigate the classic single-sample single-timescale actor-critic on continuous (infinite) state-action space, where we employ the canonical linear quadratic regulator (LQR) problem as a case study. We show that the popular single-timescale actor-critic can attain an epsilon-optimal solution with an order of epsilon to -2 sample complexity for solving LQR on the demanding continuous state-action space. Our work provides new insights into the performance of single-timescale actor-critic, which further bridges the gap between theory and practice.

Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulator

TL;DR

This paper proves that a classic single-sample single-timescale actor-critic algorithm can achieve a global -optimal policy for the continuous-state, continuous-action LQR problem, with a sample complexity of . It develops a finite-time analysis by viewing the three interconnected updates (cost estimation, critic TD, and actor natural-gradient) as an interconnected system and establishing convergence of the cost-estimator error, critic error, and gradient error at under a suitable step-size ratio . A gradient-domination property for LQR links the gradient norm to policy performance, enabling global convergence guarantees, while experiments validate improved sample efficiency over baselines. The results bridge theory and practice for AC in continuous control, highlighting how rough gradient estimates can still yield globally optimal policies with proper coupling analysis and stability constraints.

Abstract

Actor-critic methods have achieved state-of-the-art performance in various challenging tasks. However, theoretical understandings of their performance remain elusive and challenging. Existing studies mostly focus on practically uncommon variants such as double-loop or two-timescale stepsize actor-critic algorithms for simplicity. These results certify local convergence on finite state- or action-space only. We push the boundary to investigate the classic single-sample single-timescale actor-critic on continuous (infinite) state-action space, where we employ the canonical linear quadratic regulator (LQR) problem as a case study. We show that the popular single-timescale actor-critic can attain an epsilon-optimal solution with an order of epsilon to -2 sample complexity for solving LQR on the demanding continuous state-action space. Our work provides new insights into the performance of single-timescale actor-critic, which further bridges the gap between theory and practice.
Paper Structure (19 sections, 19 theorems, 212 equations, 1 figure, 2 tables, 3 algorithms)

This paper contains 19 sections, 19 theorems, 212 equations, 1 figure, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Actor-Critic Algorithms
  5. Actor-Critic for Linear Quadratic Regulator
  6. Single-sample Single-timescale Actor-Critic
  7. Main Theory
  8. Proof Sketch
  9. Experiments
  10. Conclusion and Discussion
  11. Proof of Main Theorems
  12. cost estimation error analysis
  13. Critic error analysis
  14. Natural gradient norm analysis
  15. Interconnected iteration system analysis
  16. ...and 4 more sections

Key Result

Lemma 1

For any stabilizing policy $\bm K$, the time-average cost $J(\bm K)$ and its gradient $\nabla_{\bm K} J(\bm K)$ take the following forms where $\bm{E_K}:=(\bm R+\bm{B}^\top \bm{P_KB})\bm{K}-\bm{B}^\top \bm{P_KA}$.

Figures (1)

  • Figure 1: (a) Learning results of Algorithm \ref{['alg1']}. In the figure, the cost error refers to $\frac{1}{T}\sum_{t=0}^{T-1}(\eta_t-J(\bm K_t))^2$, Critic error refers to $\frac{1}{T}\sum_{t=0}^{T-1}\Vert \bm\omega_t-\bm\omega_{\bm K_t}^\ast\Vert^2$, and the Actor error refers to $\frac{1}{T}\sum_{t=0}^{T-1}[J(\bm K_t)-J(\bm K^\ast)]$, corresponding to the conclusion in \ref{['t0']} empirically. (b) Comparison of \ref{['alg1']} with two other algorithms. The actor norm error refers to $\Vert \bm K-\bm K^* \Vert_F$. In this figure, the solid lines correspond to the mean and the shaded regions correspond to 95% confidence interval over 10 independent runs.

Theorems & Definitions (39)

  • Lemma 1: yang2019provably
  • Lemma 2: bradtke1994adaptiveyang2019provably
  • Lemma 3: Coercive Property
  • Lemma 4
  • Theorem 1
  • Lemma 5
  • Lemma 6: Upper bound for reward and feature function
  • Lemma 7: Upper bound for cost function
  • Lemma 8
  • proof
  • ...and 29 more