Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Elementary Analysis of Policy Gradient Methods

Jiacai Liu, Wenye Li, Ke Wei

TL;DR

A systematic study of policy optimization methods under the discounted MDP setting and several novel results are presented, including a new and concise local quadratic convergence rate of soft policy iteration without the assumption on the stationary distribution under the optimal policy.

Abstract

Projected policy gradient under the simplex parameterization, policy gradient and natural policy gradient under the softmax parameterization, are fundamental algorithms in reinforcement learning. There have been a flurry of recent activities in studying these algorithms from the theoretical aspect. Despite this, their convergence behavior is still not fully understood, even given the access to exact policy evaluations. In this paper, we focus on the discounted MDP setting and conduct a systematic study of the aforementioned policy optimization methods. Several novel results are presented, including 1) global linear convergence of projected policy gradient for any constant step size, 2) sublinear convergence of softmax policy gradient for any constant step size, 3) global linear convergence of softmax natural policy gradient for any constant step size, 4) global linear convergence of entropy regularized softmax policy gradient for a wider range of constant step sizes than existing result, 5) tight local linear convergence rate of entropy regularized natural policy gradient, and 6) a new and concise local quadratic convergence rate of soft policy iteration without the assumption on the stationary distribution under the optimal policy. New and elementary analysis techniques have been developed to establish these results.

Elementary Analysis of Policy Gradient Methods

TL;DR

A systematic study of policy optimization methods under the discounted MDP setting and several novel results are presented, including a new and concise local quadratic convergence rate of soft policy iteration without the assumption on the stationary distribution under the optimal policy.

Abstract

Projected policy gradient under the simplex parameterization, policy gradient and natural policy gradient under the softmax parameterization, are fundamental algorithms in reinforcement learning. There have been a flurry of recent activities in studying these algorithms from the theoretical aspect. Despite this, their convergence behavior is still not fully understood, even given the access to exact policy evaluations. In this paper, we focus on the discounted MDP setting and conduct a systematic study of the aforementioned policy optimization methods. Several novel results are presented, including 1) global linear convergence of projected policy gradient for any constant step size, 2) sublinear convergence of softmax policy gradient for any constant step size, 3) global linear convergence of softmax natural policy gradient for any constant step size, 4) global linear convergence of entropy regularized softmax policy gradient for a wider range of constant step sizes than existing result, 5) tight local linear convergence rate of entropy regularized natural policy gradient, and 6) a new and concise local quadratic convergence rate of soft policy iteration without the assumption on the stationary distribution under the optimal policy. New and elementary analysis techniques have been developed to establish these results.
Paper Structure (27 sections, 51 theorems, 332 equations, 2 figures)

This paper contains 27 sections, 51 theorems, 332 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Main Contributions
  4. Outline of This Paper
  5. General Analysis Framework
  6. Preliminaries
  7. Sublinear Convergence Analysis
  8. Global Linear Convergence Analysis
  9. Projected Policy Gradient
  10. Softmax Policy Gradient
  11. Improvement Lower and Upper Bounds
  12. Convergence Results
  13. Softmax Natural Policy Gradient
  14. Entropy Regularized Policy Gradient Methods
  15. Preliminaries
  16. ...and 12 more sections

Key Result

Lemma 2.1

Given two policies $\pi_1$ and $\pi_2$, there holds

Figures (2)

  • Figure 1: A random MDP example which shows entropy softmax PG does not converge for large step size.
  • Figure 2: Empirical justification of the local linear rate of entropy softmax NPG on random MDP with $|\mathcal{S}|=50$, $|\mathcal{A}|=20$, and $\gamma=0.99$: $\gamma$, $\gamma^{2k}$, and $\frac{1}{(1+\eta\tau)^{2k}}$ correspond to the limit convergence results presented in Theorems \ref{['theorem: ent_npg: linear convergence of log probability']}, \ref{['thm:entropyNPG-improvement']} and \ref{['thm:entropyNPG-local']}, respectively. The regularization parameter is set to $\tau=0.05$ in the test.

Theorems & Definitions (96)

  • Lemma 2.1: Performance difference lemma kakade2002approximately
  • Lemma 2.2
  • Lemma 2.3
  • Lemma 2.4: Khodadadian_Jhunjhunwala_Varma_Maguluri_2021ppgliu
  • Lemma 2.5
  • Lemma 2.6
  • Lemma 2.7
  • Theorem 2.8: Sublinear Convergence
  • proof
  • Remark 2.1
  • ...and 86 more