Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data Augmentation for Continual RL via Adversarial Gradient Episodic Memory

Sihao Wu, Xingyu Zhao, Xiaowei Huang

TL;DR

Extensive experiments show that data augmentations, such as random amplitude scaling, state-switch, mixup, adversarial augmentation, and Adv-GEM, can improve existing continual RL algorithms in terms of their average performance, catastrophic forgetting, and forward transfer, on robot control tasks.

Abstract

Data efficiency of learning, which plays a key role in the Reinforcement Learning (RL) training process, becomes even more important in continual RL with sequential environments. In continual RL, the learner interacts with non-stationary, sequential tasks and is required to learn new tasks without forgetting previous knowledge. However, there is little work on implementing data augmentation for continual RL. In this paper, we investigate the efficacy of data augmentation for continual RL. Specifically, we provide benchmarking data augmentations for continual RL, by (1) summarising existing data augmentation methods and (2) including a new augmentation method for continual RL: Adversarial Augmentation with Gradient Episodic Memory (Adv-GEM). Extensive experiments show that data augmentations, such as random amplitude scaling, state-switch, mixup, adversarial augmentation, and Adv-GEM, can improve existing continual RL algorithms in terms of their average performance, catastrophic forgetting, and forward transfer, on robot control tasks. All data augmentation methods are implemented as plug-in modules for trivial integration into continual RL methods.

Data Augmentation for Continual RL via Adversarial Gradient Episodic Memory

TL;DR

Extensive experiments show that data augmentations, such as random amplitude scaling, state-switch, mixup, adversarial augmentation, and Adv-GEM, can improve existing continual RL algorithms in terms of their average performance, catastrophic forgetting, and forward transfer, on robot control tasks.

Abstract

Data efficiency of learning, which plays a key role in the Reinforcement Learning (RL) training process, becomes even more important in continual RL with sequential environments. In continual RL, the learner interacts with non-stationary, sequential tasks and is required to learn new tasks without forgetting previous knowledge. However, there is little work on implementing data augmentation for continual RL. In this paper, we investigate the efficacy of data augmentation for continual RL. Specifically, we provide benchmarking data augmentations for continual RL, by (1) summarising existing data augmentation methods and (2) including a new augmentation method for continual RL: Adversarial Augmentation with Gradient Episodic Memory (Adv-GEM). Extensive experiments show that data augmentations, such as random amplitude scaling, state-switch, mixup, adversarial augmentation, and Adv-GEM, can improve existing continual RL algorithms in terms of their average performance, catastrophic forgetting, and forward transfer, on robot control tasks. All data augmentation methods are implemented as plug-in modules for trivial integration into continual RL methods.
Paper Structure (12 sections, 3 equations, 3 figures, 5 tables, 2 algorithms)

This paper contains 12 sections, 3 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Continual RL
  4. Data Augmentation in RL
  5. Data Augmentation for Continual RL
  6. Problem Formulation
  7. Data Augmentation Methods
  8. Data Augmentation Framework of Continual RL
  9. Experiments
  10. Experiment Setting
  11. Results
  12. Conclusion

Figures (3)

  • Figure 1: Framework for Adv-GEM data generation.
  • Figure 2: CW10 consists of 10 manipulation tasks, carefully designed to be diverse while maintaining a shared structure. This shared structure facilitates efficient continual RL.
  • Figure 3: (Top) Average Performance in MW4 for EWC and PackNet with Adv-GEM. (Bottom) The test success rate on the current task in MW4 for EWC and PackNet.