Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation

Max Qiushi Lin, Jincheng Mei, Matin Aghaei, Michael Lu, Bo Dai, Alekh Agarwal, Dale Schuurmans, Csaba Szepesvari, Sharan Vaswani

TL;DR

The paper challenges the conventional focus on approximation error in analyzing policy gradient convergence under linear function approximation, showing that global convergence can be guaranteed under feature-based conditions even when approximation error is nonzero. It introduces reward-ordering preservation and general feature conditions that ensure asymptotic global convergence in both deterministic and stochastic bandit settings, and derives convergence rates such as $O(1/T)$ in the exact/deterministic case and sublinear rates in stochastic scenarios. Moreover, it proves that Lin-SPG can achieve almost-sure global convergence with arbitrary constant learning rates, with an asymptotic average-suboptimality rate of $O(\ln(T)/T)$. These results advance the theoretical understanding of policy gradient methods with linear function approximation and suggest directions for extending the framework to general MDPs and non-linear representations.

Abstract

Policy gradient (PG) methods have played an essential role in the empirical successes of reinforcement learning. In order to handle large state-action spaces, PG methods are typically used with function approximation. In this setting, the approximation error in modeling problem-dependent quantities is a key notion for characterizing the global convergence of PG methods. We focus on Softmax PG with linear function approximation (referred to as $\texttt{Lin-SPG}$) and demonstrate that the approximation error is irrelevant to the algorithm's global convergence even for the stochastic bandit setting. Consequently, we first identify the necessary and sufficient conditions on the feature representation that can guarantee the asymptotic global convergence of $\texttt{Lin-SPG}$. Under these feature conditions, we prove that $T$ iterations of $\texttt{Lin-SPG}$ with a problem-specific learning rate result in an $O(1/T)$ convergence to the optimal policy. Furthermore, we prove that $\texttt{Lin-SPG}$ with any arbitrary constant learning rate can ensure asymptotic global convergence to the optimal policy.

Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation

TL;DR

The paper challenges the conventional focus on approximation error in analyzing policy gradient convergence under linear function approximation, showing that global convergence can be guaranteed under feature-based conditions even when approximation error is nonzero. It introduces reward-ordering preservation and general feature conditions that ensure asymptotic global convergence in both deterministic and stochastic bandit settings, and derives convergence rates such as in the exact/deterministic case and sublinear rates in stochastic scenarios. Moreover, it proves that Lin-SPG can achieve almost-sure global convergence with arbitrary constant learning rates, with an asymptotic average-suboptimality rate of . These results advance the theoretical understanding of policy gradient methods with linear function approximation and suggest directions for extending the framework to general MDPs and non-linear representations.

Abstract

Policy gradient (PG) methods have played an essential role in the empirical successes of reinforcement learning. In order to handle large state-action spaces, PG methods are typically used with function approximation. In this setting, the approximation error in modeling problem-dependent quantities is a key notion for characterizing the global convergence of PG methods. We focus on Softmax PG with linear function approximation (referred to as ) and demonstrate that the approximation error is irrelevant to the algorithm's global convergence even for the stochastic bandit setting. Consequently, we first identify the necessary and sufficient conditions on the feature representation that can guarantee the asymptotic global convergence of . Under these feature conditions, we prove that iterations of with a problem-specific learning rate result in an convergence to the optimal policy. Furthermore, we prove that with any arbitrary constant learning rate can ensure asymptotic global convergence to the optimal policy.
Paper Structure (39 sections, 43 theorems, 272 equations, 4 figures, 2 algorithms)

This paper contains 39 sections, 43 theorems, 272 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Limitations of Approximation Error in Characterizing Convergence
  4. $\texttt{Lin-SPG}$ in the Exact Setting
  5. Global Convergence is Achievable with Non-zero Approximation Error
  6. Global Convergence is Irrelevant to Non-zero Approximation Error
  7. Global Convergence: Exact Setting
  8. Warm up: Global Convergence for $K = 3$
  9. Guarantee of Global Convergence for $K \geq 3$
  10. Global Convergence: Stochastic Setting
  11. $\texttt{Lin-SPG}$ in the Stochastic Setting
  12. Decomposition of Stochastic Process
  13. Guarantee of Global Convergence
  14. Rate of Convergence
  15. Global Convergence for Arbitrary Learning Rates
  16. ...and 24 more sections

Key Result

Proposition 0

With a specific constant learning rate $\eta > 0$ and any initialization $\theta_1 \in {\mathbb{R}}^d$, alg:det_spg guarantees that $\lim_{t \to \infty} \pi_{\theta_t}(a^*) = 1$ on eg:first_example.

Figures (4)

  • Figure 1: Visualization of the optimization landscape for \ref{['eg:first_example']} (left) and \ref{['eg:second_example']} (right). These two examples share the same reward vector but have different features, which leads to different optimization landscapes. Starting at the same initialization, the red arrows demonstrate the optimization trajectories of running \ref{['alg:det_spg']} using the same learning rate. Despite both examples having similar approximation error, $\texttt{Lin-SPG}$ can converge to the optimal action in \ref{['eg:first_example']} but fails to do so in \ref{['eg:second_example']}.
  • Figure 2: The effect of feature conditions on the global convergence.
  • Figure 3: $\texttt{Lin-SPG}$ in the exact setting. The learning rate is set by \ref{['eq:step_size_for_deterministic_bandits']}. Each experiment is run on 50 randomly generated environments for $10^6$ iterations. For each environment, the features $X$ and the reward vector $r$ are randomly generated such that \ref{['assumption:reward_ordering_preservation']} is satisfied, and the features satisfy \ref{['assumption:feature_conditions_for_three_armed_linear_bandits']} when (a) $K=3$ and satisfy \ref{['assumption:general_feature_conditions']} when (b) $K=6$. $\texttt{Lin-SPG}$ converges to the optimal policy for different feature dimensions $d$, confirming the results of \ref{['theorem:three_armed_deterministic_linear_bandits', 'theorem:deterministic_linear_bandits']}.
  • Figure 4: $\texttt{Lin-SPG}$ in the stochastic setting ($K = 6$, $d = 3$) with different learning rates. We run the experiments $5$ times on each of the $5$ randomly generated environments ($25$ runs in total) for $10^6$ iterations. Each environment's underlying reward distribution is either a Bernoulli, Gaussian, or Beta distribution with a fixed mean reward vector $r \in [0, 1]^K$. For each environment, the features $X$ and the mean reward vector $r$ are randomly generated such that \ref{['assumption:no_identical_arms', 'assumption:general_feature_conditions']} are satisfied. As predicted in \ref{['theorem:stochastic_linear_bandits', 'theorem:stochastic_linear_bandits_with_arbitrary_learning_rates']}, $\texttt{Lin-SPG}$ converges to zero suboptimality within $10^6$ iterations for most of the runs, regardless of what learning rate is used.

Theorems & Definitions (51)

  • Example 1
  • Proposition 0
  • Example 2
  • Lemma 0
  • Theorem 1
  • Remark 2
  • Remark 3
  • Theorem 4
  • Remark 5
  • Lemma 5
  • ...and 41 more