Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning

Yingxiao Huo, Satya Prakash Dash, Radu Stoican, Samuel Kaski, Mingfei Sun

TL;DR

This work tackles the prohibitive cost of natural policy gradient methods in deep reinforcement learning by introducing a rank-1 inverse Fisher approximation derived from the empirical Fisher with damping. By applying the Sherman–Morrison formula in a matrix-free scheme, the update direction becomes $(F^{-1}g)$ with $F$ approximated as $ ilde F^k = frac{1}{ ext{}}\lambda I + gg^T$, enabling $O(d)$ computations per step. The authors prove global convergence under standard assumptions and provide a concrete sample-complexity bound, showing the approach scales favorably with horizon and discount factors. Empirically, the SMAC algorithm combines this update with an A2C framework and demonstrates faster convergence and stronger final performance than vanilla policy gradients, Adam-based A2C, and CG-based TRPO across classic control and MuJoCo environments, highlighting practical impact for scalable natural-gradient optimization in deep RL.

Abstract

Natural gradients have long been studied in deep reinforcement learning due to their fast convergence properties and covariant weight updates. However, computing natural gradients requires inversion of the Fisher Information Matrix (FIM) at each iteration, which is computationally prohibitive in nature. In this paper, we present an efficient and scalable natural policy optimization technique that leverages a rank-1 approximation to full inverse-FIM. We theoretically show that under certain conditions, a rank-1 approximation to inverse-FIM converges faster than policy gradients and, under some conditions, enjoys the same sample complexity as stochastic policy gradient methods. We benchmark our method on a diverse set of environments and show that it achieves superior performance to standard actor-critic and trust-region baselines.

Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning

TL;DR

This work tackles the prohibitive cost of natural policy gradient methods in deep reinforcement learning by introducing a rank-1 inverse Fisher approximation derived from the empirical Fisher with damping. By applying the Sherman–Morrison formula in a matrix-free scheme, the update direction becomes with approximated as , enabling computations per step. The authors prove global convergence under standard assumptions and provide a concrete sample-complexity bound, showing the approach scales favorably with horizon and discount factors. Empirically, the SMAC algorithm combines this update with an A2C framework and demonstrates faster convergence and stronger final performance than vanilla policy gradients, Adam-based A2C, and CG-based TRPO across classic control and MuJoCo environments, highlighting practical impact for scalable natural-gradient optimization in deep RL.

Abstract

Natural gradients have long been studied in deep reinforcement learning due to their fast convergence properties and covariant weight updates. However, computing natural gradients requires inversion of the Fisher Information Matrix (FIM) at each iteration, which is computationally prohibitive in nature. In this paper, we present an efficient and scalable natural policy optimization technique that leverages a rank-1 approximation to full inverse-FIM. We theoretically show that under certain conditions, a rank-1 approximation to inverse-FIM converges faster than policy gradients and, under some conditions, enjoys the same sample complexity as stochastic policy gradient methods. We benchmark our method on a diverse set of environments and show that it achieves superior performance to standard actor-critic and trust-region baselines.
Paper Structure (28 sections, 5 theorems, 31 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 28 sections, 5 theorems, 31 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Kronecker-factored Approximate Curvature
  4. Hessian-Free Methods
  5. Empirical Fisher Information
  6. Preliminaries
  7. Markov decision process (MDP)
  8. Performance objective and policy gradient
  9. Fisher information matrix and Natural Policy Gradient
  10. Empirical Fisher Information
  11. Sherman-Morrison formula
  12. NPG Approximation
  13. Matrix-Free Update
  14. Convergence Analysis
  15. SM-ActorCritic
  16. ...and 13 more sections

Key Result

Theorem 1

If we take the stochastic Sherman-Morrison policy update in Equation eq:sm_update and take $\eta=\frac{1}{4L_J}$, $K=\mathcal{O}\left(\frac{1}{(1-\gamma)^{2}\varepsilon^2}\right)$, $N=\mathcal{O}\left(\frac{\sigma^2}{\varepsilon^2}\right)$, and $H =\mathcal{O}\left(\log(\frac{1}{(1-\gamma)\varepsilo In total, stochastic PG samples $\mathcal{O}\left(\frac{\sigma^2}{(1-\gamma)^2\varepsilon^4}\right)

Figures (5)

  • Figure 1: Effect of Batch Size.
  • Figure 2: Results on three classic control tasks. We report the average return per episode. All results are averaged over $5$ random seeds, with shaded areas showing standard deviation.
  • Figure 3: Results on six MuJoCo tasks. We report the average return per episode. All results are averaged over $5$ random seeds, with shaded areas showing standard deviation.
  • Figure 4: Average log-probabilities of the actions taken on three classic control tasks. All results are averaged over $5$ random seeds, with shaded areas showing standard deviation.
  • Figure 5: Average log-probabilities of the actions taken on six MuJoCo tasks. All results are averaged over $5$ random seeds, with shaded areas showing standard deviation.

Theorems & Definitions (7)

  • Theorem 1
  • Remark 1
  • Remark 2
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4