Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Log-Barrier Helps Exploration in Policy Optimization

Leonardo Cesani, Matteo Papini, Marcello Restelli

Abstract

Recently, it has been shown that the Stochastic Gradient Bandit (SGB) algorithm converges to a globally optimal policy with a constant learning rate. However, these guarantees rely on unrealistic assumptions about the learning process, namely that the probability of the optimal action is always bounded away from zero. We attribute this to the lack of an explicit exploration mechanism in SGB. To address these limitations, we propose to regularize the SGB objective with a log-barrier on the parametric policy, structurally enforcing a minimal amount of exploration. We prove that Log-Barrier Stochastic Gradient Bandit (LB-SGB) matches the sample complexity of SGB, but also converges (at a slower rate) without any assumptions on the learning process. We also show a connection between the log-barrier regularization and Natural Policy Gradient, as both exploit the geometry of the policy space by controlling the Fisher information. We validate our theoretical findings through numerical simulations, showing the benefits of the log-barrier regularization.

How Log-Barrier Helps Exploration in Policy Optimization

Abstract

Recently, it has been shown that the Stochastic Gradient Bandit (SGB) algorithm converges to a globally optimal policy with a constant learning rate. However, these guarantees rely on unrealistic assumptions about the learning process, namely that the probability of the optimal action is always bounded away from zero. We attribute this to the lack of an explicit exploration mechanism in SGB. To address these limitations, we propose to regularize the SGB objective with a log-barrier on the parametric policy, structurally enforcing a minimal amount of exploration. We prove that Log-Barrier Stochastic Gradient Bandit (LB-SGB) matches the sample complexity of SGB, but also converges (at a slower rate) without any assumptions on the learning process. We also show a connection between the log-barrier regularization and Natural Policy Gradient, as both exploit the geometry of the policy space by controlling the Fisher information. We validate our theoretical findings through numerical simulations, showing the benefits of the log-barrier regularization.
Paper Structure (33 sections, 27 theorems, 150 equations, 10 figures, 11 tables, 4 algorithms)

This paper contains 33 sections, 27 theorems, 150 equations, 10 figures, 11 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Limitations of SGB
  4. Log-Barrier Stochastic Gradient Bandit
  5. Convergence of LG-SGB
  6. Convergence Analysis Assuming Bounded $c^*$
  7. Worst-case Sample Complexity
  8. Discussion
  9. Connection between Log-barrier and NPG
  10. Experimental Results
  11. Conclusions, Limitations, and Future Work
  12. Detailed Related Works
  13. Algorithmic Details
  14. Stochastic Gradient Bandit
  15. Stochastic Gradient Bandit with Entropy Regularization
  16. ...and 18 more sections

Key Result

Lemma 1

For all $\bm{\theta} \in \mathbb{R}^{K}$, and for all $\bm{r} \in \mathbb{R}^{K}$ the spectral radius of the Hessian matrix $H(\bm{\theta})\in\mathbb{R}^{K\times K}$ of $\Phi_{\eta}(\bm{\theta})$ is upper bounded by a function of $\bm{\theta}$. Precisely, for all $\bm{y}\in\mathbb{R}^{K}$,

Figures (10)

  • Figure 1: Comparison between SGB and LB-SGB with $K=\{100, 1000\}$ and $\Delta^* = 0.1$ ($100$ runs $\pm95\%$ C.I.).
  • Figure 2: Comparison between SGB and LB-SGB with $K=100$ and $\Delta^*=\{0.05, 0.005\}$ ($100$ runs $\pm95\%$ C.I.).
  • Figure 3: Comparison between LB-SGB, ENT and NPG with $K=\{10, 100\}$ and $\Delta^* = 0.1$ ($100$ runs $\pm95\%$ C.I.).
  • Figure 4: Comparison between SGB, LB-SGB and entropy-regularized SGB (ENT) with $K=10$ and different learning rates $\alpha$ ($100$ runs $\pm95\%$ C.I.).
  • Figure 5: Comparison between SGB, LB-SGB and entropy-regularized SGB (ENT) with $K=100$ and different learning rates $\alpha$ ($100$ runs $\pm95\%$ C.I.).
  • ...and 5 more figures

Theorems & Definitions (45)

  • Lemma 1: Non-uniform Smoothness
  • Lemma 2: Weak Non-Uniform Łojasiewicz
  • Lemma 3: Self-bounding Property
  • Theorem 5.1: Convergence Rate and Iteration Complexity
  • Lemma 4: Bounded $\pi_{\bt_t}(a^*)$
  • Theorem 5.2: Local Convergence
  • Theorem 5.3: Worst-case Sample Complexity
  • Corollary 1: Regret
  • Proposition 1: FIM as Covariance Matrix
  • Proposition 1: FIM as Covariance Matrix
  • ...and 35 more