$β$-DQN: Improving Deep Q-Learning By Evolving the Behavior

TL;DR

This paper tackles the exploration challenge in deep reinforcement learning by introducing β-DQN, a simple yet effective augmentation of DQN. It learns a behavior function $β(s,a)$ from replay data to create a population of diverse, temporally extended policies that balance state-action coverage, exploitation, and overestimation bias correction; an adaptive meta-controller selects the most suitable policy per episode. The method achieves broad generality and low computational overhead, outperforming several baselines on both dense and sparse reward tasks such as MiniGrid and MinAtar, while maintaining favorable performance-to-cost ratios. The work demonstrates that a general, data-driven behavior policy can facilitate robust exploration without bespoke inductive biases, and it lays groundwork for extending the approach to continuous action spaces.

Abstract

While many sophisticated exploration methods have been proposed, their lack of generality and high computational cost often lead researchers to favor simpler methods like $ε$-greedy. Motivated by this, we introduce $β$-DQN, a simple and efficient exploration method that augments the standard DQN with a behavior function $β$. This function estimates the probability that each action has been taken at each state. By leveraging $β$, we generate a population of diverse policies that balance exploration between state-action coverage and overestimation bias correction. An adaptive meta-controller is designed to select an effective policy for each episode, enabling flexible and explainable exploration. $β$-DQN is straightforward to implement and adds minimal computational overhead to the standard DQN. Experiments on both simple and challenging exploration domains show that $β$-DQN outperforms existing baseline methods across a wide range of tasks, providing an effective solution for improving exploration in deep reinforcement learning.

Figures (18)

• Figure 1: Method overview. We learn a behavior function $\beta$ from the replay memory and augment it with the $Q$ function for three purposes: state-action coverage (left), pure exploitation (middle), and overestimation correction (right). By interpolating between these strategies, we construct a policy set that balances exploration and exploitation at the intra-episodic level. A meta-controller adaptively selects a policy for each episode.
• Figure 2: (a) The first image shows the state-action pairs in memory. It implies that taking actions with low probabilities according to $\beta$ will try missing actions. The second and third images show the masked/unmasked Q values and the corresponding actions. (b) Policies with different $\delta$ and $\alpha$, demonstrating the effectiveness of our method in constructing a diverse policy set.
• Figure 3: Details during the learning. (1) The top left learning curves show that $\beta$-DQN successfully reaches the goal state, while DQN only avoids the cliff. (2) The heatmaps illustrate that $\beta$-DQN explores the entire state space efficiently. (3) The top right curves show that $\beta$-DQN initially uses $\pi_{\text{cov}}$ for state space exploration, then switches to $\pi_{\text{cor}}$ for correcting biased estimations.
• Figure 4: Policy selection varies across different tasks. In simple (LavaCrossing-Easy) or dense reward (Asterix) tasks, exploration primarily corrects estimation biases. In harder tasks (LavaCrossing-Hard), two types of exploration alternate, leading to a more complex policy selection strategy.
• Figure 5: The performance of the three basic polices. $\arg\min_a \beta$ learns nothing since it does not consider rewards. ${\arg\max}_{a:\beta>\epsilon}Q$ chooses in-sample greedy actions and performs the best. $\arg\max_a Q$ takes greedy actions among the entire action space and may take overestimated actions.

Theorems & Definitions (2)

• proposition 1
• proposition 2