Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization

Sayambhu Sen, Shalabh Bhatnagar

TL;DR

The paper tackles the sample-inefficiency of adversarial imitation learning caused by on-policy training in GAIL. It introduces an off-policy, actor–critic IL framework with a bounded tanh actor, double Q-style stabilization, and reward learning integrated into the critic, enabling stable, data-efficient learning from expert demonstrations without requiring environment rewards. The method achieves expert-level performance on the BipedalWalker-v2 task in roughly 200k environment steps, significantly reducing sample complexity compared to the GAIL baseline, though high variance during training suggests avenues for further stabilization. This approach enhances the practical viability of imitation learning for real-world robotics and control tasks where environment interactions are costly.

Abstract

Learning complex policies with Reinforcement Learning (RL) is often hindered by instability and slow convergence, a problem exacerbated by the difficulty of reward engineering. Imitation Learning (IL) from expert demonstrations bypasses this reliance on rewards. However, state-of-the-art IL methods, exemplified by Generative Adversarial Imitation Learning (GAIL)Ho et. al, suffer from severe sample inefficiency. This is a direct consequence of their foundational on-policy algorithms, such as TRPO Schulman et.al. In this work, we introduce an adversarial imitation learning algorithm that incorporates off-policy learning to improve sample efficiency. By combining an off-policy framework with auxiliary techniques specifically, double Q network based stabilization and value learning without reward function inference we demonstrate a reduction in the samples required to robustly match expert behavior.

Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization

TL;DR

The paper tackles the sample-inefficiency of adversarial imitation learning caused by on-policy training in GAIL. It introduces an off-policy, actor–critic IL framework with a bounded tanh actor, double Q-style stabilization, and reward learning integrated into the critic, enabling stable, data-efficient learning from expert demonstrations without requiring environment rewards. The method achieves expert-level performance on the BipedalWalker-v2 task in roughly 200k environment steps, significantly reducing sample complexity compared to the GAIL baseline, though high variance during training suggests avenues for further stabilization. This approach enhances the practical viability of imitation learning for real-world robotics and control tasks where environment interactions are costly.

Abstract

Learning complex policies with Reinforcement Learning (RL) is often hindered by instability and slow convergence, a problem exacerbated by the difficulty of reward engineering. Imitation Learning (IL) from expert demonstrations bypasses this reliance on rewards. However, state-of-the-art IL methods, exemplified by Generative Adversarial Imitation Learning (GAIL)Ho et. al, suffer from severe sample inefficiency. This is a direct consequence of their foundational on-policy algorithms, such as TRPO Schulman et.al. In this work, we introduce an adversarial imitation learning algorithm that incorporates off-policy learning to improve sample efficiency. By combining an off-policy framework with auxiliary techniques specifically, double Q network based stabilization and value learning without reward function inference we demonstrate a reduction in the samples required to robustly match expert behavior.

Paper Structure

This paper contains 21 sections, 30 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Previous Methods
  3. Behaviour Cloning (BC)
  4. Inverse Reinforcement Learning (IRL)
  5. Generative Adversarial Imitation Learning (GAIL)
  6. Understanding Sample Inefficiency
  7. Causes of Sample Inefficiency
  8. Methods to Improve Sample Efficiency
  9. Formulation of Algorithm Objectives
  10. Off-Policy Actor-Critic
  11. Reward Learning
  12. Value Learning
  13. Incorporating Reward Learning into the Critic
  14. Implementation Issues and Stabilization
  15. Algorithm
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Using noise sampling after output vs Using noise as input and giving a bounded action output plappert2018parameterspacenoiseexploration
  • Figure 2: Off-policy actor critic architecture for continuous control liessnerdeeprlenergy2018
  • Figure 3: BipedalWalker-v2 environment in OpenAI gym
  • Figure 4: Training performance of our off-policy imitation algorithm (blue) compared to the GAIL baseline (orange) on BipedalWalker-v2. Our method achieves expert-level rewards ( 300) significantly faster, demonstrating a substantial improvement in sample efficiency.