Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ordering-based Conditions for Global Convergence of Policy Gradient Methods

Jincheng Mei, Bo Dai, Alekh Agarwal, Mohammad Ghavamzadeh, Csaba Szepesvari, Dale Schuurmans

TL;DR

The paper tackles the challenge of understanding global convergence of policy gradient methods when using linear function approximation in a finite-arm bandit setting. It shows that global convergence can occur without realizability and that approximation error is not the right lens; instead, algorithm-specific, ordering-based conditions drive convergence. For Softmax PG, global convergence is guaranteed if the representation non-dominates and can realize a reward order via r' = X w, while for NPG, global convergence is equivalent to the least-squares projection hat_r = X (X^T X)^{-1} X^T r preserving the top action's rank, with an exponential convergence rate when satisfied. The work provides theoretical characterizations, supportive simulations, and discusses feasibility checks and extensions to MDPs and representation learning, offering a new direction beyond traditional approximation-error analyses.

Abstract

We prove that, for finite-arm bandits with linear function approximation, the global convergence of policy gradient (PG) methods depends on inter-related properties between the policy update and the representation. textcolor{blue}{First}, we establish a few key observations that frame the study: \textbf{(i)} Global convergence can be achieved under linear function approximation without policy or reward realizability, both for the standard Softmax PG and natural policy gradient (NPG). \textbf{(ii)} Approximation error is not a key quantity for characterizing global convergence in either algorithm. \textbf{(iii)} The conditions on the representation that imply global convergence are different between these two algorithms. Overall, these observations call into question approximation error as an appropriate quantity for characterizing the global convergence of PG methods under linear function approximation. \textcolor{blue}{Second}, motivated by these observations, we establish new general results: \textbf{(i)} NPG with linear function approximation achieves global convergence \emph{if and only if} the projection of the reward onto the representable space preserves the optimal action's rank, a quantity that is not strongly related to approximation error. \textbf{(ii)} The global convergence of Softmax PG occurs if the representation satisfies a non-domination condition and can preserve the ranking of rewards, which goes well beyond policy or reward realizability. We provide experimental results to support these theoretical findings.

Ordering-based Conditions for Global Convergence of Policy Gradient Methods

TL;DR

The paper tackles the challenge of understanding global convergence of policy gradient methods when using linear function approximation in a finite-arm bandit setting. It shows that global convergence can occur without realizability and that approximation error is not the right lens; instead, algorithm-specific, ordering-based conditions drive convergence. For Softmax PG, global convergence is guaranteed if the representation non-dominates and can realize a reward order via r' = X w, while for NPG, global convergence is equivalent to the least-squares projection hat_r = X (X^T X)^{-1} X^T r preserving the top action's rank, with an exponential convergence rate when satisfied. The work provides theoretical characterizations, supportive simulations, and discusses feasibility checks and extensions to MDPs and representation learning, offering a new direction beyond traditional approximation-error analyses.

Abstract

We prove that, for finite-arm bandits with linear function approximation, the global convergence of policy gradient (PG) methods depends on inter-related properties between the policy update and the representation. textcolor{blue}{First}, we establish a few key observations that frame the study: \textbf{(i)} Global convergence can be achieved under linear function approximation without policy or reward realizability, both for the standard Softmax PG and natural policy gradient (NPG). \textbf{(ii)} Approximation error is not a key quantity for characterizing global convergence in either algorithm. \textbf{(iii)} The conditions on the representation that imply global convergence are different between these two algorithms. Overall, these observations call into question approximation error as an appropriate quantity for characterizing the global convergence of PG methods under linear function approximation. \textcolor{blue}{Second}, motivated by these observations, we establish new general results: \textbf{(i)} NPG with linear function approximation achieves global convergence \emph{if and only if} the projection of the reward onto the representable space preserves the optimal action's rank, a quantity that is not strongly related to approximation error. \textbf{(ii)} The global convergence of Softmax PG occurs if the representation satisfies a non-domination condition and can preserve the ranking of rewards, which goes well beyond policy or reward realizability. We provide experimental results to support these theoretical findings.

Paper Structure

This paper contains 24 sections, 9 theorems, 100 equations, 3 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Settings
  3. The Limitations of Approximation Error in Characterizing Convergence
  4. Global Convergence is Achievable with Non-zero Approximation Error
  5. Visualization.
  6. Global Convergence is Irrelevant to Non-zero Approximation Error
  7. Global Convergence Characterization is Algorithm Dependent
  8. New Characterizations of Global Convergence for PG Methods
  9. Reward Order Preservation with Adequate Features is Sufficient for PG Convergence
  10. Intuition.
  11. Verifying reward order preservation.
  12. Main Softmax PG result.
  13. Optimal Action Preservation is Necessary and Sufficient for NPG Convergence
  14. Intuition and demonstration.
  15. Main NPG result.
  16. ...and 9 more sections

Key Result

Proposition 1

Denote $a^* \coloneqq \mathop{\mathrm{arg\,max}}\limits_{a \in [K]}{ r(a) }$. With constant $\eta > 0$ and any initialization $\theta_1 \in {\mathbb{R}}^d$, both alg:softmax_policy_gradientalg:natural_policy_gradient guarantee $\pi_{\theta_t}^\top r \to r(a^*)$ as $t \to \infty$ on eg:first_example.

Figures (3)

  • Figure 1: Visualizing the landscapes in the example problem instances.
  • Figure 2: Idea illustration.
  • Figure 3: Simulations for verifying theoretical results.

Theorems & Definitions (23)

  • Example 1
  • Proposition 1
  • Example 2
  • Example 3
  • Theorem 1: Reward order preservation, non-domination features
  • Proposition 2
  • Corollary 1: Linearly realizable rewards, non-domination features
  • Lemma 1
  • Remark 1
  • Lemma 2: Non-negative covariance of order preservation
  • ...and 13 more