Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mirror Learning: A Unifying Framework of Policy Optimisation

Jakub Grudzien Kuba, Christian Schroeder de Witt, Jakob Foerster

TL;DR

A novel theoretical framework, named Mirror Learning, is introduced, which provides theoretical guarantees to a large class of algorithms, including TRPO and PPO, which suggests that the empirical performance of state-of-the-art methods is a direct consequence of their theoretical properties, rather than of aforementioned approximate analogies.

Abstract

Modern deep reinforcement learning (RL) algorithms are motivated by either the generalised policy iteration (GPI) or trust-region learning (TRL) frameworks. However, algorithms that strictly respect these theoretical frameworks have proven unscalable. Surprisingly, the only known scalable algorithms violate the GPI/TRL assumptions, e.g. due to required regularisation or other heuristics. The current explanation of their empirical success is essentially "by analogy": they are deemed approximate adaptations of theoretically sound methods. Unfortunately, studies have shown that in practice these algorithms differ greatly from their conceptual ancestors. In contrast, in this paper we introduce a novel theoretical framework, named Mirror Learning, which provides theoretical guarantees to a large class of algorithms, including TRPO and PPO. While the latter two exploit the flexibility of our framework, GPI and TRL fit in merely as pathologically restrictive corner cases thereof. This suggests that the empirical performance of state-of-the-art methods is a direct consequence of their theoretical properties, rather than of aforementioned approximate analogies. Mirror learning sets us free to boldly explore novel, theoretically sound RL algorithms, a thus far uncharted wonderland.

Mirror Learning: A Unifying Framework of Policy Optimisation

TL;DR

A novel theoretical framework, named Mirror Learning, is introduced, which provides theoretical guarantees to a large class of algorithms, including TRPO and PPO, which suggests that the empirical performance of state-of-the-art methods is a direct consequence of their theoretical properties, rather than of aforementioned approximate analogies.

Abstract

Modern deep reinforcement learning (RL) algorithms are motivated by either the generalised policy iteration (GPI) or trust-region learning (TRL) frameworks. However, algorithms that strictly respect these theoretical frameworks have proven unscalable. Surprisingly, the only known scalable algorithms violate the GPI/TRL assumptions, e.g. due to required regularisation or other heuristics. The current explanation of their empirical success is essentially "by analogy": they are deemed approximate adaptations of theoretically sound methods. Unfortunately, studies have shown that in practice these algorithms differ greatly from their conceptual ancestors. In contrast, in this paper we introduce a novel theoretical framework, named Mirror Learning, which provides theoretical guarantees to a large class of algorithms, including TRPO and PPO. While the latter two exploit the flexibility of our framework, GPI and TRL fit in merely as pathologically restrictive corner cases thereof. This suggests that the empirical performance of state-of-the-art methods is a direct consequence of their theoretical properties, rather than of aforementioned approximate analogies. Mirror learning sets us free to boldly explore novel, theoretically sound RL algorithms, a thus far uncharted wonderland.
Paper Structure (22 sections, 6 theorems, 72 equations, 3 figures)

This paper contains 22 sections, 6 theorems, 72 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Preliminaries
  4. Generalised Policy Iteration
  5. Trust-Region Learning
  6. Mirror Learning
  7. The Framework
  8. Mirror Learning View of RL Phenomena
  9. Generalised Policy Iteration
  10. Trust-Region Learning
  11. TRPO
  12. PPO
  13. Related Work
  14. Graph-theoretical Interpretation
  15. Numerical Experiments
  16. ...and 7 more sections

Key Result

Lemma 3.0

Let $\pi_{\text{old}}$ and $\pi_{\text{new}}$ be policies. Suppose that Then, $\pi_{\text{new}}$ is better than $\pi_{\text{old}}$, so that for every state $s$,

Figures (3)

  • Figure 1: Known RL frameworks and algorithms as points in the infinite space of theoretically sound mirror learning algorithms.
  • Figure 2: An intuitive view on the policy DAG and initial steps of mirror learning. A policy vertex has a neighbour, within its neighbourhood, which improves the return.
  • Figure 3: Mirror learning algorithms with different drifts and neighbourhood operators tested on simple environments. The solid lines represent the return, and the dotted ones represent the total drift. Algorithms in the left column use the drift ball neighbourhood, while those in the right use the KL ball. In both columns there are algorithms with each of the aforementioned drifts. Results are for one seed per environment and algorithm.

Theorems & Definitions (15)

  • Definition 3.0
  • Definition 3.0
  • Lemma 3.0
  • Definition 3.0
  • Lemma 3.0
  • Theorem 3.1: The Fundamental Theorem of Mirror Learning
  • Definition 6.0: Policy DAG
  • Definition 1.0
  • Definition 1.0
  • Lemma 2.0
  • ...and 5 more