Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DQN Performance with Epsilon Greedy Policies and Prioritized Experience Replay

Daniel Perkins, Oscar J. Escobar, Luke Green

TL;DR

This work analyzes how epsilon-greedy exploration schedules and replay memory affect Deep Q-Network (DQN) training in a finite, deterministic CartPole setting. It contrasts classic Q-learning with DQN, examining exponential and other decaying epsilon strategies, and evaluates uniform versus prioritized experience replay. Key findings show that fast, super-linear epsilon decay can improve cumulative rewards, while prioritized replay often boosts sample efficiency but can introduce runtime and performance variability depending on the task and schedule. The study highlights the interplay between exploration, memory management, and function approximation, offering practical guidance for robust deep RL in resource-constrained environments. The results suggest that PER is beneficial in more complex domains, but for simple environments like CartPole, uniform replay can be sufficient, with hyperparameter tuning being crucial for optimal performance.

Abstract

We present a detailed study of Deep Q-Networks in finite environments, emphasizing the impact of epsilon-greedy exploration schedules and prioritized experience replay. Through systematic experimentation, we evaluate how variations in epsilon decay schedules affect learning efficiency, convergence behavior, and reward optimization. We investigate how prioritized experience replay leads to faster convergence and higher returns and show empirical results comparing uniform, no replay, and prioritized strategies across multiple simulations. Our findings illuminate the trade-offs and interactions between exploration strategies and memory management in DQN training, offering practical recommendations for robust reinforcement learning in resource-constrained settings.

DQN Performance with Epsilon Greedy Policies and Prioritized Experience Replay

TL;DR

This work analyzes how epsilon-greedy exploration schedules and replay memory affect Deep Q-Network (DQN) training in a finite, deterministic CartPole setting. It contrasts classic Q-learning with DQN, examining exponential and other decaying epsilon strategies, and evaluates uniform versus prioritized experience replay. Key findings show that fast, super-linear epsilon decay can improve cumulative rewards, while prioritized replay often boosts sample efficiency but can introduce runtime and performance variability depending on the task and schedule. The study highlights the interplay between exploration, memory management, and function approximation, offering practical guidance for robust deep RL in resource-constrained environments. The results suggest that PER is beneficial in more complex domains, but for simple environments like CartPole, uniform replay can be sufficient, with hyperparameter tuning being crucial for optimal performance.

Abstract

We present a detailed study of Deep Q-Networks in finite environments, emphasizing the impact of epsilon-greedy exploration schedules and prioritized experience replay. Through systematic experimentation, we evaluate how variations in epsilon decay schedules affect learning efficiency, convergence behavior, and reward optimization. We investigate how prioritized experience replay leads to faster convergence and higher returns and show empirical results comparing uniform, no replay, and prioritized strategies across multiple simulations. Our findings illuminate the trade-offs and interactions between exploration strategies and memory management in DQN training, offering practical recommendations for robust reinforcement learning in resource-constrained settings.

Paper Structure

This paper contains 19 sections, 6 equations, 10 figures.

Table of Contents

  1. Background & Motivation
  2. Q-Learning
  3. RL Challenges
  4. Exploration-Exploitation-Trade off
  5. Credit Assignment Problem
  6. Deep Reinforcement Learning
  7. Assumptions & Objective.
  8. Simulation Environment
  9. Q-Learning
  10. Deep Q-Network: Epsilon Greedy Algorithm/Policy
  11. Theory
  12. Exponential Decay
  13. Other Decaying Epsilon Schedules
  14. Deep Q-Network: Experience Replay
  15. No Experience Replay.
  16. ...and 4 more sections

Figures (10)

  • Figure 1: A representation of the RL framework as first introduced by sutton2018reinforcement. The agent finding itself in state $S_t$ and having received reward $R_t$ takes action $A_{t+1}$ and transitions into new state $S_{t+1}$ and receives a reward $R_{t+1}$. The cycle then begins again.
  • Figure 2: The Cart Pole Environment
  • Figure 3: The rewards obtained using an exponential decay $0.9999^t$ for $\varepsilon$-greedy in normal Q-learning. Compare to Fig. \ref{['fig:exponentialdecay']}.
  • Figure 4: Cumulative reward with exponential epsilon decay for various values of $\beta$
  • Figure 5: Cumulative reward with various epsilon decay schedules
  • ...and 5 more figures