Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weak Convergence Analysis of Online Neural Actor-Critic Algorithms

Samuel Chun-Hei Lam, Justin Sirignano, Ziheng Wang

TL;DR

The paper addresses online neural actor-critic methods in reinforcement learning where the data distribution evolves with policy updates. It develops a two-timescale NTK-based framework and uses Poisson equations to handle fluctuations from non-i.i.d. samples, proving that the time-rescaled actor and critic converge to a nonlinear random ODE as the width $N$ and steps grow. The limit dynamics imply the critic converges to the true value function for the current policy and the actor converges to a stationary point, with the gradient vanishing and the policy gradient becoming asymptotically unbiased. This provides a rigorous foundation for the convergence of online neural actor-critic methods and highlights the role of ergodicity and stochastic fluctuations in shaping the limit behavior, offering insights into long-run policy optimization in online settings.

Abstract

We prove that a single-layer neural network trained with the online actor critic algorithm converges in distribution to a random ordinary differential equation (ODE) as the number of hidden units and the number of training steps $\rightarrow \infty$. In the online actor-critic algorithm, the distribution of the data samples dynamically changes as the model is updated, which is a key challenge for any convergence analysis. We establish the geometric ergodicity of the data samples under a fixed actor policy. Then, using a Poisson equation, we prove that the fluctuations of the model updates around the limit distribution due to the randomly-arriving data samples vanish as the number of parameter updates $\rightarrow \infty$. Using the Poisson equation and weak convergence techniques, we prove that the actor neural network and critic neural network converge to the solutions of a system of ODEs with random initial conditions. Analysis of the limit ODE shows that the limit critic network will converge to the true value function, which will provide the actor an asymptotically unbiased estimate of the policy gradient. We then prove that the limit actor network will converge to a stationary point.

Weak Convergence Analysis of Online Neural Actor-Critic Algorithms

TL;DR

The paper addresses online neural actor-critic methods in reinforcement learning where the data distribution evolves with policy updates. It develops a two-timescale NTK-based framework and uses Poisson equations to handle fluctuations from non-i.i.d. samples, proving that the time-rescaled actor and critic converge to a nonlinear random ODE as the width and steps grow. The limit dynamics imply the critic converges to the true value function for the current policy and the actor converges to a stationary point, with the gradient vanishing and the policy gradient becoming asymptotically unbiased. This provides a rigorous foundation for the convergence of online neural actor-critic methods and highlights the role of ergodicity and stochastic fluctuations in shaping the limit behavior, offering insights into long-run policy optimization in online settings.

Abstract

We prove that a single-layer neural network trained with the online actor critic algorithm converges in distribution to a random ordinary differential equation (ODE) as the number of hidden units and the number of training steps . In the online actor-critic algorithm, the distribution of the data samples dynamically changes as the model is updated, which is a key challenge for any convergence analysis. We establish the geometric ergodicity of the data samples under a fixed actor policy. Then, using a Poisson equation, we prove that the fluctuations of the model updates around the limit distribution due to the randomly-arriving data samples vanish as the number of parameter updates . Using the Poisson equation and weak convergence techniques, we prove that the actor neural network and critic neural network converge to the solutions of a system of ODEs with random initial conditions. Analysis of the limit ODE shows that the limit critic network will converge to the true value function, which will provide the actor an asymptotically unbiased estimate of the policy gradient. We then prove that the limit actor network will converge to a stationary point.
Paper Structure (31 sections, 23 theorems, 272 equations, 1 algorithm)

This paper contains 31 sections, 23 theorems, 272 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Convergence Analysis of Actor-critic Algorithms
  3. ODE Methods
  4. Finite time analysis
  5. Our Mathematical Approach
  6. Organisation of the analysis
  7. Actor-Critic Algorithms
  8. Markov Decision Processes
  9. Policy in the MDP
  10. Online Neural Network Actor-critic Algorithm
  11. Step 1: Update of the critic network:
  12. Step 2: Update of the actor network:
  13. Main Result
  14. Derivation of the limit ODEs
  15. Evolution of the Pre-limit Processes
  16. ...and 16 more sections

Key Result

Theorem 3.3

Let Assumptions as:activation_function and as:NN_condition hold, and let the learning rate for the critic parameter updates be $\alpha^{N}=\alpha/N$ for an $\alpha > 0$. Then, the process $\left(\mu_{t}^{N}, \nu_t^N, P_{t}^{N}, Q_t^N \right)$ converges weakly in the space $D_E([0, T])$ as $N \righta where $\mathcal{G}, \mathcal{H}$ are the weak limits of $P^N_0$ and $Q^N_0$, which are mean-zero Ga

Theorems & Definitions (51)

  • Definition 2.1: Markov decision process (MDP)
  • Definition 2.3: State and action-value functions
  • Remark 2.4
  • Definition 2.5: State and state-action visiting measures, see e.g. policygradient1999konda2002actor and Section 2 of wang2021global
  • Remark 2.6
  • Remark 2.8
  • Theorem 3.3
  • Lemma 3.4
  • Theorem 3.5
  • Remark 3.6
  • ...and 41 more