Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs

Michael Lu, Matin Aghaei, Anant Raj, Sharan Vaswani

TL;DR

These techniques result in a theoretically-principled PG algorithm that does not require explicit exploration, the knowledge of the reward gap, the reward distributions, or the noise, and are empirically compared to PG approaches that require oracle knowledge, and demonstrate competitive performance.

Abstract

We consider (stochastic) softmax policy gradient (PG) methods for bandits and tabular Markov decision processes (MDPs). While the PG objective is non-concave, recent research has used the objective's smoothness and gradient domination properties to achieve convergence to an optimal policy. However, these theoretical results require setting the algorithm parameters according to unknown problem-dependent quantities (e.g. the optimal action or the true reward vector in a bandit problem). To address this issue, we borrow ideas from the optimization literature to design practical, principled PG methods in both the exact and stochastic settings. In the exact setting, we employ an Armijo line-search to set the step-size for softmax PG and demonstrate a linear convergence rate. In the stochastic setting, we utilize exponentially decreasing step-sizes, and characterize the convergence rate of the resulting algorithm. We show that the proposed algorithm offers similar theoretical guarantees as the state-of-the art results, but does not require the knowledge of oracle-like quantities. For the multi-armed bandit setting, our techniques result in a theoretically-principled PG algorithm that does not require explicit exploration, the knowledge of the reward gap, the reward distributions, or the noise. Finally, we empirically compare the proposed methods to PG approaches that require oracle knowledge, and demonstrate competitive performance.

Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs

TL;DR

These techniques result in a theoretically-principled PG algorithm that does not require explicit exploration, the knowledge of the reward gap, the reward distributions, or the noise, and are empirically compared to PG approaches that require oracle knowledge, and demonstrate competitive performance.

Abstract

We consider (stochastic) softmax policy gradient (PG) methods for bandits and tabular Markov decision processes (MDPs). While the PG objective is non-concave, recent research has used the objective's smoothness and gradient domination properties to achieve convergence to an optimal policy. However, these theoretical results require setting the algorithm parameters according to unknown problem-dependent quantities (e.g. the optimal action or the true reward vector in a bandit problem). To address this issue, we borrow ideas from the optimization literature to design practical, principled PG methods in both the exact and stochastic settings. In the exact setting, we employ an Armijo line-search to set the step-size for softmax PG and demonstrate a linear convergence rate. In the stochastic setting, we utilize exponentially decreasing step-sizes, and characterize the convergence rate of the resulting algorithm. We show that the proposed algorithm offers similar theoretical guarantees as the state-of-the art results, but does not require the knowledge of oracle-like quantities. For the multi-armed bandit setting, our techniques result in a theoretically-principled PG algorithm that does not require explicit exploration, the knowledge of the reward gap, the reward distributions, or the noise. Finally, we empirically compare the proposed methods to PG approaches that require oracle knowledge, and demonstrate competitive performance.
Paper Structure (47 sections, 67 theorems, 152 equations, 7 figures, 3 tables)

This paper contains 47 sections, 67 theorems, 152 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Setup & Background
  3. Policy Gradient in the Exact Setting
  4. Policy Gradient in the Stochastic Setting
  5. Stochastic Softmax Policy Gradient
  6. Theoretical Convergence
  7. Faster Rates
  8. Experimental Evaluation
  9. Discussion
  10. Definitions
  11. Proofs in \ref{['section:pg_wo_entropy']}
  12. Proof Of Theorem \ref{['theorem:pg_line_search']}
  13. Proof Of Theorem \ref{['theorem:pg_line_search_exp_new']}
  14. Additional Lemmas
  15. Proofs in \ref{['section:spg_wo_entropy']}
  16. ...and 32 more sections

Key Result

theorem 1

Assuming $f$ is (i) $L$-smooth, (ii) satisfies the non-uniform Ł ojasiewciz condition with $\xi = 0$, and (iii) $\mu := \inf_{t \geq 1} [C({\theta_t})]^2 > 0$, using update:dpg and Armijo line-search to set the step-size results in the following convergence: where $h \in (0, 1)$ and $\eta_{\max}$ is the upper-bound on the step-size.

Figures (7)

  • Figure 1: Comparing softmax PG that (i) uses a step-size that satisfies the Armijo condition in \ref{['eq:armijo']} (denoted as PG-LS), (ii) uses a step-size that satisfies the Armijo condition in \ref{['eq:trans_line_search']} (PG-Log-LS) to GNPG (GNPG), PG-A (PG-A) and PG with a fixed step-size (PG) in the tabular MDP setting.
  • Figure 2: Expected sub-optimality gap across various environments. SPG-ESS and SPG-ESS [D] is comparable to SPG-O-G and SPG-O-R without using any oracle-like knowledge of the environment.
  • Figure 3: Softmax PG, on-policy stochastic gradient
  • Figure 4: Sub-optimality gap across various environments and initializations. Top Row: the initial policy's parameters is uniform, i.e. $\theta_0(a) = 0 \quad \forall a$. Bottom Row: the initial policy's parameters is "bad", i.e. $\theta_0(a') = 12$ where $a' = \mathop{\mathrm{arg\,min}}\limits_a r(a)$
  • Figure 5: Expected sub-optimality gap across various environments with uniform initialization
  • ...and 2 more figures

Theorems & Definitions (107)

  • theorem 1
  • theorem 2
  • theorem 3
  • theorem 4
  • theorem 5
  • corollary 1
  • theorem 5
  • proof
  • corollary 2
  • proof
  • ...and 97 more