Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimality

François Ged, Maria Han Veiga

TL;DR

It is shown that the optimal policy of the infinite horizon max-entropy objective can be approximated arbitrarily well by the optimal policy of the MPG framework, and a criterion for global optimality when the policy is parametrized by a neural network in terms of the neural tangent kernel at convergence.

Abstract

A novel Policy Gradient (PG) algorithm, called $\textit{Matryoshka Policy Gradient}$ (MPG), is introduced and studied, in the context of fixed-horizon max-entropy reinforcement learning, where an agent aims at maximizing entropy bonuses additional to its cumulative rewards. In the linear function approximation setting with softmax policies, we prove uniqueness and characterize the optimal policy of the entropy regularized objective, together with global convergence of MPG. These results are proved in the case of continuous state and action space. MPG is intuitive, theoretically sound and we furthermore show that the optimal policy of the infinite horizon max-entropy objective can be approximated arbitrarily well by the optimal policy of the MPG framework. Finally, we provide a criterion for global optimality when the policy is parametrized by a neural network in terms of the neural tangent kernel at convergence. As a proof of concept, we evaluate numerically MPG on standard test benchmarks.

Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimality

TL;DR

It is shown that the optimal policy of the infinite horizon max-entropy objective can be approximated arbitrarily well by the optimal policy of the MPG framework, and a criterion for global optimality when the policy is parametrized by a neural network in terms of the neural tangent kernel at convergence.

Abstract

A novel Policy Gradient (PG) algorithm, called (MPG), is introduced and studied, in the context of fixed-horizon max-entropy reinforcement learning, where an agent aims at maximizing entropy bonuses additional to its cumulative rewards. In the linear function approximation setting with softmax policies, we prove uniqueness and characterize the optimal policy of the entropy regularized objective, together with global convergence of MPG. These results are proved in the case of continuous state and action space. MPG is intuitive, theoretically sound and we furthermore show that the optimal policy of the infinite horizon max-entropy objective can be approximated arbitrarily well by the optimal policy of the MPG framework. Finally, we provide a criterion for global optimality when the policy is parametrized by a neural network in terms of the neural tangent kernel at convergence. As a proof of concept, we evaluate numerically MPG on standard test benchmarks.
Paper Structure (45 sections, 18 theorems, 115 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 45 sections, 18 theorems, 115 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Comparison with previous results and approaches.
  4. Fixed-horizon max-entropy RL
  5. Definitions
  6. Markov Decision Process
  7. Value and Q functions.
  8. Objective and optimal policy
  9. Matryoshka Policy Gradient
  10. Policy parametrization
  11. Definition of the MPG update
  12. Global convergence: the realizable case
  13. Intuition of the proof: the bandit case.
  14. Global convergence: beyond the realizability assumption
  15. Projectional consistency property
  16. ...and 30 more sections

Key Result

Proposition 1

There exists a unique uniformly optimal policy, denoted by $\pi_*=(\pi_*^{(1)},\ldots,\pi_*^{(n)})\in\mathcal{P}_n$. The $i$-step optimal policies, $i=1,\ldots,n$, can be obtained as follows: for all $a\in\mathcal{A}$, $s\in\mathcal{S}$, where $Q_*^{(i+1)}$ is a short-hand notation for $Q_{\pi_*}^{(i+1)}$ recursively defined as in eq: gen case definition Q-function.

Figures (3)

  • Figure 1: Analytical task. Convergence of 5 agents with random initialisation during training; the errors are measured through the $L_\infty$-norm and $\pi^{(1)}$ denotes the one-step policy and $\pi^{(2)}$ the two-step policy. On the left, the convergence of the found 1-step and 2-step policies towards the optimal policies when the parametric space can represent the policy (i.e. when assumption \ref{['assumption: optimal policy in RKHS']} holds) is shown. On the right, the convergence of the 1-step and 2-step policies towards the optimal projected policies (i.e. when assumption \ref{['assumption: optimal policy in RKHS']} does not hold) is shown.
  • Figure 2: Frozen Lake. Left: Cumulative rewards per episode during training time when training using different RL algorithms with the best found set of hyper-parameters. Right: Cumulative rewards per episode after training, each trained agent attempts to solve the task $100$ times.
  • Figure 3: Cart Pole. Left: Cumulative rewards per episode during training time when training using different RL algorithms with the best found set of hyper-parameters. Right: Cumulative rewards per episode after training, each trained agent attempts to solve the task $100$ times.

Theorems & Definitions (34)

  • Proposition 1
  • Lemma 1
  • Proposition 2
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Proposition 3
  • Corollary 1
  • Remark 1
  • Lemma 2
  • ...and 24 more