Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory

Tianjiao Luo, Tim Pearce, Huayu Chen, Jianfei Chen, Jun Zhu

TL;DR

A control-theoretic analysis of GAIL is conducted and a novel controller is derived that not only pushes GAIL to the desired equilibrium but also achieves asymptotic stability in a 'one-step' setting.

Abstract

Generative Adversarial Imitation Learning (GAIL) trains a generative policy to mimic a demonstrator. It uses on-policy Reinforcement Learning (RL) to optimize a reward signal derived from a GAN-like discriminator. A major drawback of GAIL is its training instability - it inherits the complex training dynamics of GANs, and the distribution shift introduced by RL. This can cause oscillations during training, harming its sample efficiency and final policy performance. Recent work has shown that control theory can help with the convergence of a GAN's training. This paper extends this line of work, conducting a control-theoretic analysis of GAIL and deriving a novel controller that not only pushes GAIL to the desired equilibrium but also achieves asymptotic stability in a 'one-step' setting. Based on this, we propose a practical algorithm 'Controlled-GAIL' (C-GAIL). On MuJoCo tasks, our controlled variant is able to speed up the rate of convergence, reduce the range of oscillation and match the expert's distribution more closely both for vanilla GAIL and GAIL-DAC.

C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory

TL;DR

A control-theoretic analysis of GAIL is conducted and a novel controller is derived that not only pushes GAIL to the desired equilibrium but also achieves asymptotic stability in a 'one-step' setting.

Abstract

Generative Adversarial Imitation Learning (GAIL) trains a generative policy to mimic a demonstrator. It uses on-policy Reinforcement Learning (RL) to optimize a reward signal derived from a GAN-like discriminator. A major drawback of GAIL is its training instability - it inherits the complex training dynamics of GANs, and the distribution shift introduced by RL. This can cause oscillations during training, harming its sample efficiency and final policy performance. Recent work has shown that control theory can help with the convergence of a GAN's training. This paper extends this line of work, conducting a control-theoretic analysis of GAIL and deriving a novel controller that not only pushes GAIL to the desired equilibrium but also achieves asymptotic stability in a 'one-step' setting. Based on this, we propose a practical algorithm 'Controlled-GAIL' (C-GAIL). On MuJoCo tasks, our controlled variant is able to speed up the rate of convergence, reduce the range of oscillation and match the expert's distribution more closely both for vanilla GAIL and GAIL-DAC.
Paper Structure (23 sections, 12 theorems, 50 equations, 4 figures, 5 tables, 1 algorithm)

This paper contains 23 sections, 12 theorems, 50 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Problem Setting
  5. Dynamical Systems and Control Theory
  6. Analyzing GAIL as a Dynamical System
  7. GAIL Dynamics
  8. On the Convergence of GAIL
  9. Controlled GAIL
  10. Controlling the Training Process of GAIL
  11. Analyzing the Stability of Controlled GAIL
  12. A Practical Method to Stabilize GAIL
  13. Evaluation
  14. Experimental Setup
  15. Results
  16. ...and 8 more sections

Key Result

Theorem 2.5

(Principle of Linearized Stability)la1976stability A controlled dynamical system (dynamic_control) with equilibrium $\Bar{x}$ is asymptotically stable if all eigenvalues of $\mathbb{J}(f(\Bar{x}) + u(t))$ have negative real parts, where $\mathbb{J} (f(\Bar{x}) + u(t))$ represents the Jacobian of $f(

Figures (4)

  • Figure 1: Normalized return curves for controlled GAIL-DAC with four expert demonstrations on five MuJoCo environments averaged over five random seeds. The x-axis represents the number of gradient step updates in millions and the y-axis represents the normalized environment reward, where 1 stands for the expert policy return and 0 stands for the random policy return
  • Figure 2: State Wasserstein distance (lower is better) between expert and learned policies, over number of gradient step updates. Our controlled variant matches the expert distribution more closely.
  • Figure 3: Number of gradient step updates (in millions) required to reach $95\%$ of the max-return for various numbers of expert trajectories on MuJoCo environments averaged over five random seeds.
  • Figure 4: Normalized returns curves for controlled GAIL with $k = 0.1$, $k = 1$, and $k = 10$ on MuJoCo environments, where on the y-axis, 1 represents expert policy return and 0 represents random policy return

Theorems & Definitions (24)

  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Definition 2.4
  • Theorem 2.5
  • Corollary 2.6
  • Theorem 3.1
  • Definition 3.2
  • Theorem 3.3
  • Theorem 4.1
  • ...and 14 more