Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Study of Continual Learning Methods for Q-Learning

Benedikt Bagus, Alexander Gepperth

TL;DR

An empirical comparison of selected CL methods in a RL problem where a physically simulated robot must follow a racetrack by vision shows that dedicated CL methods can significantly improve learning when compared to the baseline technique of “experience replay”.

Abstract

We present an empirical study on the use of continual learning (CL) methods in a reinforcement learning (RL) scenario, which, to the best of our knowledge, has not been described before. CL is a very active recent research topic concerned with machine learning under non-stationary data distributions. Although this naturally applies to RL, the use of dedicated CL methods is still uncommon. This may be due to the fact that CL methods often assume a decomposition of CL problems into disjoint sub-tasks of stationary distribution, that the onset of these sub-tasks is known, and that sub-tasks are non-contradictory. In this study, we perform an empirical comparison of selected CL methods in a RL problem where a physically simulated robot must follow a racetrack by vision. In order to make CL methods applicable, we restrict the RL setting and introduce non-conflicting subtasks of known onset, which are however not disjoint and whose distribution, from the learner's point of view, is still non-stationary. Our results show that dedicated CL methods can significantly improve learning when compared to the baseline technique of "experience replay".

A Study of Continual Learning Methods for Q-Learning

TL;DR

An empirical comparison of selected CL methods in a RL problem where a physically simulated robot must follow a racetrack by vision shows that dedicated CL methods can significantly improve learning when compared to the baseline technique of “experience replay”.

Abstract

We present an empirical study on the use of continual learning (CL) methods in a reinforcement learning (RL) scenario, which, to the best of our knowledge, has not been described before. CL is a very active recent research topic concerned with machine learning under non-stationary data distributions. Although this naturally applies to RL, the use of dedicated CL methods is still uncommon. This may be due to the fact that CL methods often assume a decomposition of CL problems into disjoint sub-tasks of stationary distribution, that the onset of these sub-tasks is known, and that sub-tasks are non-contradictory. In this study, we perform an empirical comparison of selected CL methods in a RL problem where a physically simulated robot must follow a racetrack by vision. In order to make CL methods applicable, we restrict the RL setting and introduce non-conflicting subtasks of known onset, which are however not disjoint and whose distribution, from the learner's point of view, is still non-stationary. Our results show that dedicated CL methods can significantly improve learning when compared to the baseline technique of "experience replay".
Paper Structure (20 sections, 2 equations, 9 figures, 7 tables)

This paper contains 20 sections, 2 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Catastrophic forgetting in RL
  3. Related work: CL approaches
  4. Related work: Continual RL
  5. Mapping CL to RL
  6. Goals and contribution
  7. Methods
  8. CL approaches
  9. Models
  10. RL Sub-tasks
  11. Benchmarking
  12. Metrics
  13. Experiments
  14. Baselines
  15. Results
  16. ...and 5 more sections

Figures (9)

  • Figure 1: Schematic representation of a RL control loop, which enables decision-making.
  • Figure 2: Illustration of environment shifts considered in this article. A simulated robot is trained to follow the black line in a succession of environments: A) straight line B) straight line + left curve C) straight line + left/right curve. Please note that all of these scenarios include concept drift as well from the point of view of the learner, due to the exploration of the state-action space.
  • Figure 3: The simulated robot, which closely models the popular $3\pi$ robot.
  • Figure 4: All $\Sigma_{e}$s of an outstanding A-GEM run. Data points are scaled by episode length and colored by the average. Sub-task boundaries are marked as vertical lines.
  • Figure 5: The agent's deviation from the line, evaluated before training.
  • ...and 4 more figures