On the Reduction of Variance and Overestimation of Deep Q-Learning

TL;DR

The paper tackles instability in deep Q-learning caused by variance and overestimation of target values. It proposes a simple, compatible modification by applying dropout to the DQN architecture to stabilize training and reduce bias. Through experiments on CARTPOLE and Gridworld, dropout-DQN demonstrates decreased variance and diminished overestimation, along with improved policy performance. The approach is readily extensible to DQN variants and could enhance robustness in more complex environments.

Abstract

The breakthrough of deep Q-Learning on different types of environments revolutionized the algorithmic design of Reinforcement Learning to introduce more stable and robust algorithms, to that end many extensions to deep Q-Learning algorithm have been proposed to reduce the variance of the target values and the overestimation phenomena. In this paper, we examine new methodology to solve these issues, we propose using Dropout techniques on deep Q-Learning algorithm as a way to reduce variance and overestimation. We also present experiments conducted on benchmark environments, demonstrating the effectiveness of our methodology in enhancing stability and reducing both variance and overestimation in model performance.

Figures (6)

• Figure 1: Some proposed Dropout methods for different neural networks architectures.
• Figure 2: Markov Decision Process (Famous RL framework).
• Figure 3: Dropout DQN with different Dropout methods in CARTPOLE environment. The bold lines represent the average scores obtained over 10 independent learning trials, while the shaded areas indicate the range of the standard deviation.
• Figure 4: Gridworld problem. The agent starts its journey from the upper-right corner of the grid, while a reward of +1 is positioned in the bottom-left corner.
• Figure 5: Average Expected value in Gridworld. Dropout methods on DQN lead to less overestimation (positive-bias). The lines are averages over 50 independent learning trials. The dotted line represents the Optimal Policy.