Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise

Li Meng, Morten Goodwin, Anis Yazidi, Paal Engelstad

TL;DR

This work addresses the degradation of diversity in Bootstrapped DQN by replacing fixed priors with Gaussian noise added to target values during updates, scaled by $scale = 1 + beta * max_a Q_theta^A(s,a)$. The proposed method preserves the multi-head ensemble structure while introducing per-update noise to broaden exploration without architectural changes. Empirical results on 49 Atari games show that Boot-DQN+NP yields higher maximal and mean evaluation scores in the majority of games, improving convergence and enabling better plateau escape. The findings demonstrate that target-value noise can effectively enhance diversity and performance in deep Q-learning, with potential for integration with more advanced exploration strategies and world-model approaches.

Abstract

Q-learning is one of the most well-known Reinforcement Learning algorithms. There have been tremendous efforts to develop this algorithm using neural networks. Bootstrapped Deep Q-Learning Network is amongst them. It utilizes multiple neural network heads to introduce diversity into Q-learning. Diversity can sometimes be viewed as the amount of reasonable moves an agent can take at a given state, analogous to the definition of the exploration ratio in RL. Thus, the performance of Bootstrapped Deep Q-Learning Network is deeply connected with the level of diversity within the algorithm. In the original research, it was pointed out that a random prior could improve the performance of the model. In this article, we further explore the possibility of replacing priors with noise and sample the noise from a Gaussian distribution to introduce more diversity into this algorithm. We conduct our experiment on the Atari benchmark and compare our algorithm to both the original and other related algorithms. The results show that our modification of the Bootstrapped Deep Q-Learning algorithm achieves significantly higher evaluation scores across different types of Atari games. Thus, we conclude that replacing priors with noise can improve Bootstrapped Deep Q-Learning's performance by ensuring the integrity of diversities.

Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise

TL;DR

This work addresses the degradation of diversity in Bootstrapped DQN by replacing fixed priors with Gaussian noise added to target values during updates, scaled by . The proposed method preserves the multi-head ensemble structure while introducing per-update noise to broaden exploration without architectural changes. Empirical results on 49 Atari games show that Boot-DQN+NP yields higher maximal and mean evaluation scores in the majority of games, improving convergence and enabling better plateau escape. The findings demonstrate that target-value noise can effectively enhance diversity and performance in deep Q-learning, with potential for integration with more advanced exploration strategies and world-model approaches.

Abstract

Q-learning is one of the most well-known Reinforcement Learning algorithms. There have been tremendous efforts to develop this algorithm using neural networks. Bootstrapped Deep Q-Learning Network is amongst them. It utilizes multiple neural network heads to introduce diversity into Q-learning. Diversity can sometimes be viewed as the amount of reasonable moves an agent can take at a given state, analogous to the definition of the exploration ratio in RL. Thus, the performance of Bootstrapped Deep Q-Learning Network is deeply connected with the level of diversity within the algorithm. In the original research, it was pointed out that a random prior could improve the performance of the model. In this article, we further explore the possibility of replacing priors with noise and sample the noise from a Gaussian distribution to introduce more diversity into this algorithm. We conduct our experiment on the Atari benchmark and compare our algorithm to both the original and other related algorithms. The results show that our modification of the Bootstrapped Deep Q-Learning algorithm achieves significantly higher evaluation scores across different types of Atari games. Thus, we conclude that replacing priors with noise can improve Bootstrapped Deep Q-Learning's performance by ensuring the integrity of diversities.
Paper Structure (6 sections, 4 equations, 4 figures, 3 tables, 2 algorithms)

This paper contains 6 sections, 4 equations, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Method
  3. Experimental Details
  4. Results
  5. Discussion
  6. Conclusion

Figures (4)

  • Figure 1: The structure of our NN. It takes the same input and have the same parameters of convolutional layers as in DQN, and then branches into K heads. Each head contains a fully connected layer, followed by an output layer.
  • Figure 2: The performance profiles of 49 Atari games up to $\tau=8$. The blue line is the percentage of the score larger than $\tau$ using noise. The orange line is the percentage of that without using noise.
  • Figure 3: The evaluation results of playing 10 Atari games. The blue lines are the mean evaluation scores of using noise. The orange lines are the mean evaluation scores without using noise. The 95% confidence intervals are also plotted in addition to the scores. All are evaluated every 250000 steps.
  • Figure 4: The Maximal Q-values corresponding to Fig. \ref{['fig:res1']}. Maximal Q-values are updated each time the network gets updated. Q-values are from the predictions of the policy network. Those values are used to update the parameter $scale$.