Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator

Haoyu Han, Heng Yang

TL;DR

This work reveals that the NSR of the REINFORCE policy-gradient estimator is highly non-uniform across parameter space and can blow up near optimal policies, offering a concrete mechanism for instability in stochastic policy optimization. The authors provide exact NSR characterizations for finite-horizon linear-quadratic-Gaussian (LQG) systems (one-step and multi-step) and exact moment-based evaluation for polynomial dynamics, along with an upper bound for general nonlinear dynamics with expressive Gaussian policies. A lifted-dynamics framework and Gaussian moment techniques (Isserlis) enable closed-form NSR expressions or exact numerical procedures, clarifying how NSR scales with initial-state covariance, policy covariance, and horizon. Experiments across linear, polynomial, and nonlinear settings show NSR typically increases as optimization nears the optimum, sometimes exploding and triggering instability or policy collapse, thereby motivating variance-reduction and exploration-aware strategies for robust RL. These results illuminate the exploration–exploitation tension in REINFORCE and guide principled design choices for maintaining informative gradient estimates during learning.

Abstract

Policy-gradient methods are widely used in reinforcement learning, yet training often becomes unstable or slows down as learning progresses. We study this phenomenon through the noise-to-signal ratio (NSR) of a policy-gradient estimator, defined as the estimator variance (noise) normalized by the squared norm of the true gradient (signal). Our main result is that, for (i) finite-horizon linear systems with Gaussian policies and linear state-feedback, and (ii) finite-horizon polynomial systems with Gaussian policies and polynomial feedback, the NSR of the REINFORCE estimator can be characterized exactly-either in closed form or via numerical moment-evaluation algorithms-without approximation. For general nonlinear dynamics and expressive policies (including neural policies), we further derive a general upper bound on the variance. These characterizations enable a direct examination of how NSR varies across policy parameters and how it evolves along optimization trajectories (e.g. SGD and Adam). Across a range of examples, we find that the NSR landscape is highly non-uniform and typically increases as the policy approaches an optimum; in some regimes it blows up, which can trigger training instability and policy collapse.

Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator

TL;DR

This work reveals that the NSR of the REINFORCE policy-gradient estimator is highly non-uniform across parameter space and can blow up near optimal policies, offering a concrete mechanism for instability in stochastic policy optimization. The authors provide exact NSR characterizations for finite-horizon linear-quadratic-Gaussian (LQG) systems (one-step and multi-step) and exact moment-based evaluation for polynomial dynamics, along with an upper bound for general nonlinear dynamics with expressive Gaussian policies. A lifted-dynamics framework and Gaussian moment techniques (Isserlis) enable closed-form NSR expressions or exact numerical procedures, clarifying how NSR scales with initial-state covariance, policy covariance, and horizon. Experiments across linear, polynomial, and nonlinear settings show NSR typically increases as optimization nears the optimum, sometimes exploding and triggering instability or policy collapse, thereby motivating variance-reduction and exploration-aware strategies for robust RL. These results illuminate the exploration–exploitation tension in REINFORCE and guide principled design choices for maintaining informative gradient estimates during learning.

Abstract

Policy-gradient methods are widely used in reinforcement learning, yet training often becomes unstable or slows down as learning progresses. We study this phenomenon through the noise-to-signal ratio (NSR) of a policy-gradient estimator, defined as the estimator variance (noise) normalized by the squared norm of the true gradient (signal). Our main result is that, for (i) finite-horizon linear systems with Gaussian policies and linear state-feedback, and (ii) finite-horizon polynomial systems with Gaussian policies and polynomial feedback, the NSR of the REINFORCE estimator can be characterized exactly-either in closed form or via numerical moment-evaluation algorithms-without approximation. For general nonlinear dynamics and expressive policies (including neural policies), we further derive a general upper bound on the variance. These characterizations enable a direct examination of how NSR varies across policy parameters and how it evolves along optimization trajectories (e.g. SGD and Adam). Across a range of examples, we find that the NSR landscape is highly non-uniform and typically increases as the policy approaches an optimum; in some regimes it blows up, which can trigger training instability and policy collapse.
Paper Structure (37 sections, 11 theorems, 271 equations, 6 figures)

This paper contains 37 sections, 11 theorems, 271 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Linear Systems, Linear Feedback
  3. NSR of the One-step REINFORCE Estimator
  4. NSR of the Multi-step REINFORCE Estimator
  5. Experiments
  6. Polynomial Systems, Polynomial Feedback
  7. Nonlinear Systems, Generic Feedback
  8. Conclusions
  9. Related Work
  10. Proof of Lemma \ref{['lem:lqg-policy-score']}
  11. Proof of Lemma \ref{['lem:IS_shorthand']}
  12. Proof of Theorem \ref{['thm:lqg-variance-one-step']}
  13. Proof of Theorem \ref{['thm:lqr-lift']}
  14. Proof of Theorem \ref{['thm:lqr-variance-T-compose']}
  15. Second moment expansions.
  16. ...and 22 more sections

Key Result

Lemma 1

For a linear Gaussian policy $a_t \sim \mathcal{N}(K s_t, \Sigma)$ with $\Sigma=\text{diag}(e^{2\ell})$, the score-function gradients are where $\odot$ is the Hadamard product, $\mathbf 1\in\mathbb R^m$ denotes the all-ones vector, and $\varepsilon_t = a_t - K s_t\sim\mathcal{N}(0,\Sigma)$.

Figures (6)

  • Figure 1: NSR of the REINFORCE estimator in one-step LQG with isotropic $\Sigma=\sigma^2 I$ and $\Sigma_0=\sigma_0^2 I$ for double integrator \ref{['eq:double-integrator']}.
  • Figure 2: NSR and objective along optimization trajectories on a double-integrator system with $T=30$. Top: optimizer trajectories overlaid on the NSR landscape. Bottom: learning curves (objective vs. iteration). The three optimizers (GD, SGD, Adam) start from the same initial policy (square), move toward the optimal policy (star), and terminate at triangles. The NSR increases markedly as the policy approaches optimality, consistent with our theory. As the NSR grows, SGD and Adam exhibit oscillations in the learning curves.
  • Figure 3: Variance growth as a function of the horizon $T$ for four 2D systems (1D control) with varying $\rho(F)$. We fix the initial-state covariance at $\Sigma_0 = I_2$ and the policy covariance at $\Sigma = 0.1 I_1$.
  • Figure 4: NSR (bottom) and objective (top) during training on the quadratic system. As the policy oscillates near the optimum, a single inaccurate gradient step triggers policy collapse.
  • Figure 5: NSR and objective along optimization trajectories on polynomial systems. (a): quadratic system; (b): cubic system. Top: optimizer trajectories and NSR landscape; Bottom: learning curves. The NSR increases when approaching the optimal policy same as linear systems. GD consistently approaches the optimum while SGD and Adam may get stuck in high-NSR regions.
  • ...and 1 more figures

Theorems & Definitions (22)

  • Lemma 1
  • Lemma 2: Gaussian quadratic-product shorthand
  • Theorem 3: Variance and gradients for one-step LQG
  • Theorem 4: Lifted $T$-step system
  • Theorem 5: Variance and gradients of multi-step LQG
  • Corollary 6: Stationary points of $J(K,\Sigma)$
  • Theorem 7: Multi-step LQG variance bound
  • Theorem 8: Spectral norm of lifted state map
  • Proposition 9: Polynomial form of REINFORCE and exact Gaussian-moment evaluation
  • Theorem 10: Upper bounds for generic nonlinear system
  • ...and 12 more