Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Primacy Bias in Deep Reinforcement Learning

Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, Aaron Courville

TL;DR

Deep RL models can overfit to initial interactions, a primacy bias that degrades later learning. The authors propose a lightweight resetting mechanism that reinitializes the last layers of the agent while keeping the replay buffer, which consistently improves performance across Atari 100k and DeepMind Control Suite and enables more aggressive replay and longer $n$-step targets. The work combines empirical demonstrations with analysis of how replay ratio, target length, and TD failure modes interact with resets, and provides practical guidelines for implementing resets with minimal overhead. Overall, resetting emerges as a simple yet powerful regularization tool that facilitates better learning dynamics and broader hyperparameter exploration in deep RL.

Abstract

This work identifies a common flaw of deep reinforcement learning (RL) algorithms: a tendency to rely on early interactions and ignore useful evidence encountered later. Because of training on progressively growing datasets, deep RL agents incur a risk of overfitting to earlier experiences, negatively affecting the rest of the learning process. Inspired by cognitive science, we refer to this effect as the primacy bias. Through a series of experiments, we dissect the algorithmic aspects of deep RL that exacerbate this bias. We then propose a simple yet generally-applicable mechanism that tackles the primacy bias by periodically resetting a part of the agent. We apply this mechanism to algorithms in both discrete (Atari 100k) and continuous action (DeepMind Control Suite) domains, consistently improving their performance.

The Primacy Bias in Deep Reinforcement Learning

TL;DR

Deep RL models can overfit to initial interactions, a primacy bias that degrades later learning. The authors propose a lightweight resetting mechanism that reinitializes the last layers of the agent while keeping the replay buffer, which consistently improves performance across Atari 100k and DeepMind Control Suite and enables more aggressive replay and longer -step targets. The work combines empirical demonstrations with analysis of how replay ratio, target length, and TD failure modes interact with resets, and provides practical guidelines for implementing resets with minimal overhead. Overall, resetting emerges as a simple yet powerful regularization tool that facilitates better learning dynamics and broader hyperparameter exploration in deep RL.

Abstract

This work identifies a common flaw of deep reinforcement learning (RL) algorithms: a tendency to rely on early interactions and ignore useful evidence encountered later. Because of training on progressively growing datasets, deep RL agents incur a risk of overfitting to earlier experiences, negatively affecting the rest of the learning process. Inspired by cognitive science, we refer to this effect as the primacy bias. Through a series of experiments, we dissect the algorithmic aspects of deep RL that exacerbate this bias. We then propose a simple yet generally-applicable mechanism that tackles the primacy bias by periodically resetting a part of the agent. We apply this mechanism to algorithms in both discrete (Atari 100k) and continuous action (DeepMind Control Suite) domains, consistently improving their performance.
Paper Structure (32 sections, 18 figures, 6 tables)

This paper contains 32 sections, 18 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. The Primacy Bias
  4. Heavy Priming Causes Unrecoverable Overfitting
  5. Experiences of Primed Agents are Sufficient
  6. Have You Tried Resetting It?
  7. Experiments
  8. Setup
  9. Resets Consistently Improve Performance
  10. Learning Dynamics of Agents with Resets
  11. Elements Behind the Success of Resets
  12. Replay ratio
  13. $n$-step targets
  14. TD failure modes
  15. What and how to reset
  16. ...and 17 more sections

Figures (18)

  • Figure 1: Undiscounted returns on quadruped-run for SAC with and without heavy priming on the first $100$ transitions. An agent extremely affected by the primacy bias is unable to learn even after collecting hundreds of thousands of new transitions. Mean and std are estimated over 10 runs.
  • Figure 2: Undiscounted returns on quadruped-run for SAC trained with $9$ updates per step. SAC failing is a standard agent; SAC with failing agent buffer is an agent initialized with the replay buffer of the first agent, which allows it to learn quickly. Mean and std are estimated over 10 runs.
  • Figure 3: Point estimates and 95% bootstrap confidence intervals for the performance of SPR with resets and prior methods on Atari 100k. Results for SPR and SPR + resets are over 20 seeds per game; others are taken from agarwal2021deep and use 100 seeds. The additional baselines contextualize the numerical impact of resets. Per-environment results are available in Appendix \ref{['sec:per_env']}.
  • Figure 4: Four examples showing diverse effects of resets for SAC (32 updates per step, resetting every $2 \times 10^5$ steps) on DMC tasks. After each reset, performance recovers quickly due to keeping the replay buffer. In cheetah-run, the baseline agent consistently succeeds at the task and resets provide no major benefit. In all other tasks, resets increase performance and often reduce variance. Mean and std are estimated over 10 runs.
  • Figure 5: Performance of SAC (left) and SPR (right) and with and without resets for different replay ratios and a fixed default $n$. The right-hand plots visualize the percent improvement gained by adding resets. Agents with higher replay ratio are more prone to the primacy bias and hence benefit more from mitigating it.
  • ...and 13 more figures