HAVER: Instance-Dependent Error Bounds for Maximum Mean Estimation and Applications to Q-Learning and Monte Carlo Tree Search
Tuan Ngo Nguyen, Jay Barrett, Kwang-Sung Jun
TL;DR
This work tackles the problem of estimating the maximum mean μ1 across K distributions from passive samples, a task central to reinforcement learning and search-based planning. It introduces HAVER, a novel Head Averaging estimator, and proves an instance-dependent MSE bound that matches the oracle rate and, in several regimes, surpasses it when many arms are near-optimal. The paper also derives concrete corollaries for equal-sample, all-arms-equal, and near-top-arm scenarios, showing accelerations up to 1/(K N) in some cases. Empirical results across multi-armed bandits, Q-learning, and MCTS demonstrate HAVER’s superior performance over established estimators, confirming its practical value for improving sample efficiency in RL and tree-search contexts.
Abstract
We study the problem of estimating the \emph{value} of the largest mean among K distributions via samples from them (rather than estimating \emph{which} distribution has the largest mean), which arises from various machine learning tasks including Q-learning and Monte Carlo Tree Search (MCTS). While there have been a few proposed algorithms, their performance analyses have been limited to their biases rather than a precise error metric. In this paper, we propose a novel algorithm called HAVER (Head AVERaging) and analyze its mean squared error. Our analysis reveals that HAVER has a compelling performance in two respects. First, HAVER estimates the maximum mean as well as the oracle who knows the identity of the best distribution and reports its sample mean. Second, perhaps surprisingly, HAVER exhibits even better rates than this oracle when there are many distributions near the best one. Both of these improvements are the first of their kind in the literature, and we also prove that the naive algorithm that reports the largest empirical mean does not achieve these bounds. Finally, we confirm our theoretical findings via numerical experiments where we implement HAVER in bandit, Q-learning, and MCTS algorithms. In these experiments, HAVER consistently outperforms the baseline methods, demonstrating its effectiveness across different applications.