Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geometry and convergence of natural policy gradient methods

Johannes Müller, Guido Montúfar

TL;DR

This work analyzes convergence of natural policy gradient methods for finite-state, finite-action MDPs with regular policy parametrizations, showing that state-action trajectories follow gradient flows under Hessian geometries of regularizers such as conditional entropy and entropy. By viewing Kakade's and Morimura's NPG as instantiations of Hessian natural gradients in state-action space, the authors establish global convergence and explicit rates: linear convergence for unregularized and regularized flows under these geometries, as well as sublinear rates for beta-divergence–based NPG. The analysis further reveals a discrete-time interpretation of regularized NPG as an inexact Newton method, yielding local quadratic convergence when the step size equals the regularization strength. Experiments corroborate the rates and illustrate the tightness of the theory, highlighting the practical relevance for planning problems where exact gradients are available. Overall, the work provides a unified Hessian-geometry framework for diverse NPG methods, connecting continuous-flow dynamics to discrete updates and offering sharp convergence guarantees and insights for regularized policy optimization in MDPs.$

Abstract

We study the convergence of several natural policy gradient (NPG) methods in infinite-horizon discounted Markov decision processes with regular policy parametrizations. For a variety of NPGs and reward functions we show that the trajectories in state-action space are solutions of gradient flows with respect to Hessian geometries, based on which we obtain global convergence guarantees and convergence rates. In particular, we show linear convergence for unregularized and regularized NPG flows with the metrics proposed by Kakade and Morimura and co-authors by observing that these arise from the Hessian geometries of conditional entropy and entropy respectively. Further, we obtain sublinear convergence rates for Hessian geometries arising from other convex functions like log-barriers. Finally, we interpret the discrete-time NPG methods with regularized rewards as inexact Newton methods if the NPG is defined with respect to the Hessian geometry of the regularizer. This yields local quadratic convergence rates of these methods for step size equal to the penalization strength.

Geometry and convergence of natural policy gradient methods

TL;DR

This work analyzes convergence of natural policy gradient methods for finite-state, finite-action MDPs with regular policy parametrizations, showing that state-action trajectories follow gradient flows under Hessian geometries of regularizers such as conditional entropy and entropy. By viewing Kakade's and Morimura's NPG as instantiations of Hessian natural gradients in state-action space, the authors establish global convergence and explicit rates: linear convergence for unregularized and regularized flows under these geometries, as well as sublinear rates for beta-divergence–based NPG. The analysis further reveals a discrete-time interpretation of regularized NPG as an inexact Newton method, yielding local quadratic convergence when the step size equals the regularization strength. Experiments corroborate the rates and illustrate the tightness of the theory, highlighting the practical relevance for planning problems where exact gradients are available. Overall, the work provides a unified Hessian-geometry framework for diverse NPG methods, connecting continuous-flow dynamics to discrete updates and offering sharp convergence guarantees and insights for regularized policy optimization in MDPs.$

Abstract

We study the convergence of several natural policy gradient (NPG) methods in infinite-horizon discounted Markov decision processes with regular policy parametrizations. For a variety of NPGs and reward functions we show that the trajectories in state-action space are solutions of gradient flows with respect to Hessian geometries, based on which we obtain global convergence guarantees and convergence rates. In particular, we show linear convergence for unregularized and regularized NPG flows with the metrics proposed by Kakade and Morimura and co-authors by observing that these arise from the Hessian geometries of conditional entropy and entropy respectively. Further, we obtain sublinear convergence rates for Hessian geometries arising from other convex functions like log-barriers. Finally, we interpret the discrete-time NPG methods with regularized rewards as inexact Newton methods if the NPG is defined with respect to the Hessian geometry of the regularizer. This yields local quadratic convergence rates of these methods for step size equal to the penalization strength.
Paper Structure (35 sections, 20 theorems, 79 equations, 5 figures)

This paper contains 35 sections, 20 theorems, 79 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. Notation
  5. Markov decision processes
  6. Natural gradients
  7. Definition and general properties of natural gradients
  8. Natural gradient as best improvement direction
  9. Choice of a geometry in model space
  10. Invariance axiomatic geometries
  11. Hessian geometries
  12. Local Hessian of Bregman divergences
  13. Connection to Gauss-Newton method
  14. Natural policy gradient methods
  15. Policy gradients
  16. ...and 20 more sections

Key Result

Proposition 1

The set $\mathcal{N}$ of state-action frequencies is a polytope given by $\mathcal{N} = \Delta_{\mathcal{S}\times\mathcal{A}}\cap\mathcal{L} = \mathbb R_{\ge0}^{\mathcal{S}\times\mathcal{A}}\cap\mathcal{L}$, where and $\ell_s(\eta) \coloneqq \sum_{a} \eta_{sa} - \gamma\sum_{s',a'} \eta_{s'a'}\alpha(s|s', a') - (1-\gamma) \mu_s$.

Figures (5)

  • Figure 1: Schematic drawing of parametric models with an objective function $\ell$ and resulting parameter objective function $L$; note that neither the choice of geometry in the model space nor the factorization or the model space itself is uniquely determined by the objective function $L$.
  • Figure 2: Transition graph and reward of the MDP example.
  • Figure 3: State-action trajectories for different PG methods, which are vanilla PG, Kakade's NPG and $\sigma$-NPG, where Morimura's NPG corresponds to $\sigma=1$; the state-action polytope is shown in gray inside a three dimensional projection of the the simplex $\Delta_{\mathcal{S}\times\mathcal{A}}$; shown are trajectories with the same random $30$ initial values for every method; the maximizer $\eta^\ast$ is located at the upper left corner of the state-action polytope.
  • Figure 4: Plots of the trajectories of the individual methods inside the policy polytope $\Delta_\mathcal{A}^\mathcal{S}\cong[0,1]^2$; additionally, a heatmap of the reward function $\pi\mapsto R(\pi)$ is shown; the maximizer $\pi^\ast$ is located at the upper left corner of the policy polytope.
  • Figure 5: Plot of the optimality gaps $R^\ast-R(\theta(t))$ during optimization; note that for vanilla PG and $\sigma>1$ these are log-log plots since we expect a decay like $t^{-1}$ and $t^{-1/(\sigma-1)}$ respectively, which are shown as a dashed gray line; Kakade's and Morimura's NPG are at a log plot since we expect a linear convergence; finally, for $\sigma<1$ we observe finite time convergence.

Theorems & Definitions (50)

  • Proposition 1: State-action polytope of MDPs, derman1970finite
  • Proposition 3: Inverse of state-action map, mueller2022geometry
  • Definition 4: General natural gradient
  • Theorem 5: Natural gradient leads to steepest descent in model space
  • Example 6: Hessian geometries
  • Theorem 7: Policy gradient theorem
  • Definition 8: Regular policy parametrization
  • Definition 9: Kakade's NPG and geometry in policy space
  • Remark 10
  • Theorem 11: Kakade's geometry as conditional entropy Hessian geometry
  • ...and 40 more