Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Effects and Uncertainty Quantification in Neural Actor Critic Algorithms

Nikos Georgoudios, Konstantinos Spiliopoulos, Justin Sirignano

TL;DR

This work derives an asymptotic expansion of the network outputs, interpreted as statistical estimators, in order to clarify their structure, and shows that the variance decays as a power of the network width, implying improved statistical robustness as the scaling parameter approaches one.

Abstract

We investigate the neural Actor Critic algorithm using shallow neural networks for both the Actor and Critic models. The focus of this work is twofold: first, to compare the convergence properties of the network outputs under various scaling schemes as the network width and the number of training steps tend to infinity; and second, to provide precise control of the approximation error associated with each scaling regime. Previous work has shown convergence to ordinary differential equations with random initial conditions under inverse square root scaling in the network width. In this work, we shift the focus from convergence speed alone to a more comprehensive statistical characterization of the algorithm's output, with the goal of quantifying uncertainty in neural Actor Critic methods. Specifically, we study a general inverse polynomial scaling in the network width, with an exponent treated as a tunable hyperparameter taking values strictly between one half and one. We derive an asymptotic expansion of the network outputs, interpreted as statistical estimators, in order to clarify their structure. To leading order, we show that the variance decays as a power of the network width, with an exponent equal to one half minus the scaling parameter, implying improved statistical robustness as the scaling parameter approaches one. Numerical experiments support this behavior and further suggest faster convergence for this choice of scaling. Finally, our analysis yields concrete guidelines for selecting algorithmic hyperparameters, including learning rates and exploration rates, as functions of the network width and the scaling parameter, ensuring provably favorable statistical behavior.

Scaling Effects and Uncertainty Quantification in Neural Actor Critic Algorithms

TL;DR

This work derives an asymptotic expansion of the network outputs, interpreted as statistical estimators, in order to clarify their structure, and shows that the variance decays as a power of the network width, implying improved statistical robustness as the scaling parameter approaches one.

Abstract

We investigate the neural Actor Critic algorithm using shallow neural networks for both the Actor and Critic models. The focus of this work is twofold: first, to compare the convergence properties of the network outputs under various scaling schemes as the network width and the number of training steps tend to infinity; and second, to provide precise control of the approximation error associated with each scaling regime. Previous work has shown convergence to ordinary differential equations with random initial conditions under inverse square root scaling in the network width. In this work, we shift the focus from convergence speed alone to a more comprehensive statistical characterization of the algorithm's output, with the goal of quantifying uncertainty in neural Actor Critic methods. Specifically, we study a general inverse polynomial scaling in the network width, with an exponent treated as a tunable hyperparameter taking values strictly between one half and one. We derive an asymptotic expansion of the network outputs, interpreted as statistical estimators, in order to clarify their structure. To leading order, we show that the variance decays as a power of the network width, with an exponent equal to one half minus the scaling parameter, implying improved statistical robustness as the scaling parameter approaches one. Numerical experiments support this behavior and further suggest faster convergence for this choice of scaling. Finally, our analysis yields concrete guidelines for selecting algorithmic hyperparameters, including learning rates and exploration rates, as functions of the network width and the scaling parameter, ensuring provably favorable statistical behavior.
Paper Structure (42 sections, 48 theorems, 353 equations, 2 figures)

This paper contains 42 sections, 48 theorems, 353 equations, 2 figures.

Table of Contents

  1. Introduction
  2. The Actor-Critic Neural Network Model and Assumptions
  3. Main Results
  4. Leading order convergence: the zeroth order term
  5. Next order convergence: first order fluctuations
  6. Completing the asymptotic series expansion.
  7. Numerical Results
  8. Experimental Study of the Bias
  9. Experimental Study of the Variance
  10. Leading Order Convergence
  11. Pre-Limit Equations
  12. Analysis of the Martingale Terms and of the Initial Condition
  13. Analysis of the Actor Model Output, Policy, and Stationary Measure Convergence
  14. Analysis of the Empirical Measure Process Convergence
  15. Proof of Theorem \ref{['leading_order_conv_th']}: Convergence of the Actor and Critic Network
  16. ...and 27 more sections

Key Result

Theorem 3.1

Let the assumptions in section assumptions_sec hold. Let $n\in \mathbb{N}$ and let $\beta \in \left( \frac{2n-1}{2n}, \frac{2n+1}{2n+2} \right)$. Then the following expansions hold for the neural network outputs $P_t^N$ and $Q_t^N$, as $N\rightarrow\infty$ where the functions $P_t^{(i)}$ and $Q_t^{(i)}$ are solutions of ODEs with zero initial condition for $i<n$, see Q_P_higher_order_error_terms_l

Figures (2)

  • Figure 1: The Reward and Actor MSE Loss as a function of training time.
  • Figure 2: Standard deviation Monte Carlo estimates for the Actor, the Critic, and the Rewards as a function of training progress for different values of the scaling parameter $\beta$.

Theorems & Definitions (78)

  • Theorem 3.1
  • Remark 3.2
  • Theorem 3.3
  • Definition 3.4
  • Proposition 3.5
  • Proposition 3.6
  • Theorem 3.7
  • Definition 3.8
  • Remark 3.9
  • Proposition 3.10
  • ...and 68 more