Addressing Maximization Bias in Reinforcement Learning with Two-Sample Testing

TL;DR

This work reframes overestimation bias in reinforcement learning as a maximum expectation estimation problem and introduces two statistically grounded estimators, the T-Estimator and the K-Estimator, based on two-sample testing and kernel smoothing. TE provides a tunable bias-variance tradeoff via a significance level alpha, while KE generalizes this with a family of kernels, preserving variance bounds and computational efficiency. The estimators are integrated into both tabular and deep Q-learning frameworks (including BDQN), with convergence guarantees in the tabular case and an adaptive variant (Ada-TE-BDQN) to minimize absolute bias during training. Empirical results across internet ads, Cliff Walking, and MinAtar show that TE/KE can achieve robust performance by controlling bias without sacrificing stability, highlighting the value of adaptive bias management in RL and guiding future extensions to continuous actions and exploration interactions.

Abstract

Value-based reinforcement-learning algorithms have shown strong results in games, robotics, and other real-world applications. Overestimation bias is a known threat to those algorithms and can sometimes lead to dramatic performance decreases or even complete algorithmic failure. We frame the bias problem statistically and consider it an instance of estimating the maximum expected value (MEV) of a set of random variables. We propose the $T$-Estimator (TE) based on two-sample testing for the mean, that flexibly interpolates between over- and underestimation by adjusting the significance level of the underlying hypothesis tests. We also introduce a generalization, termed $K$-Estimator (KE), that obeys the same bias and variance bounds as the TE and relies on a nearly arbitrary kernel function. We introduce modifications of $Q$-Learning and the Bootstrapped Deep $Q$-Network (BDQN) using the TE and the KE, and prove convergence in the tabular setting. Furthermore, we propose an adaptive variant of the TE-based BDQN that dynamically adjusts the significance level to minimize the absolute estimation bias. All proposed estimators and algorithms are thoroughly tested and validated on diverse tasks and environments, illustrating the bias control and performance potential of the TE and KE.

Figures (24)

• Figure 1: Comparison of the ME, DE, and TE with level of significance in parentheses.
• Figure 1: Original kernel functions and optimized specification for minimizing the squared bias in Figures \ref{['fig:MSE_TE']} and \ref{['fig:MSE_KE']}.
• Figure 1: Algorithm comparison on Asterix for learning rates $\tau \in \{10^{-5}, 10^{-4}\}$. The first two rows show the return and bias over time for $\tau = 10^{-5}$, while the results for $\tau=10^{-4}$ are displayed in rows three and four. Regarding algorithms, the left column includes the DQN, DDQN, and SCDQN; the middle column displays the MaxMin DQN, BDQN, and two TE-BDQNs; and the right column contains the remaining TE-BDQNs, the KE-BDQN, and the Ada-TE-BDQN results. The peak of the bias curve of the BDQN in row four of column two is at approximately 50, which we do not display to ensure the readability of the other curves.
• Figure 2: Comparison of the ME, DE, and KE with kernel in parentheses.
• Figure 2: Bias, variance, and MSE for different estimators of the MEV in the case of two Gaussian random variables using the optimized TE and KE's shown in Figure \ref{['fig:Kernel_funcs']}. The optimized beta kernel opts for a similar solution as the optimized TE.

Theorems & Definitions (16)

• Lemma 1
• proof
• Lemma 2
• Corollary 1
• proof
• Corollary 2
• Theorem 5.1
• proof
• Corollary 3
• proof