Investigating Scale Independent UCT Exploration Factor Strategies

TL;DR

The paper addresses the sensitivity of UCT to reward scale by evaluating scale-aware, adaptive exploration strategies for the UCT constant $λ$ across a broad suite of MCTS tasks. It introduces Global Std, a scale-invariant approach where $λ = C·σ$ with $σ$ the empirical standard deviation of all state-action $Q$ values, and demonstrates its superior generalization and peak performance over fixed $λ$ and other strategies. Across 17 MDPs and 11 two-player games, Global Std with $C=2$ consistently outperforms alternatives, suggesting a practical drop-in replacement for Vanilla UCT with broad applicability. The work also analyzes theoretical properties, compares local/global sampling strategies, and provides extensive supplementary data to support reproducibility and future extensions in adaptive exploration for MCTS.

Abstract

The Upper Confidence Bounds For Trees (UCT) algorithm is not agnostic to the reward scale of the game it is applied to. For zero-sum games with the sparse rewards of $\{-1,0,1\}$ at the end of the game, this is not a problem, but many games often feature dense rewards with hand-picked reward scales, causing a node's Q-value to span different magnitudes across different games. In this paper, we evaluate various strategies for adaptively choosing the UCT exploration constant $λ$, called $λ$-strategies, that are agnostic to the game's reward scale. These $λ$-strategies include those proposed in the literature as well as five new strategies. Given our experimental results, we recommend using one of our newly suggested $λ$-strategies, which is to choose $λ$ as $2 \cdot σ$ where $σ$ is the empirical standard deviation of all state-action pairs' Q-values of the search tree. This method outperforms existing $λ$-strategies across a wide range of tasks both in terms of a single parameter value and the peak performances obtained by optimizing all available parameters.

Figures (9)

• Figure 1: A visualization of which state-action pairs are taken into consideration when determining the exploration factor $\lambda$ for the bottom-left state-action pair that is marked in red. The four images all depict a search tree with five nodes where arrows represent deterministic actions. The state-action pairs used to determine the corresponding exploration constant all intersect a red ellipse. Each subfigure's caption lists the $\lambda$-strategies that use the marked set of state-action pairs for the $\lambda$ calculation.
• Figure 2: The normalized pairings score for all considered $\lambda$-strategies. The score was constructed by considering all iteration budgets and all environments. Per pairing, the maximum performance over all $C$ values (i.e. $\{0.125,0.25,0.5,1,2,4,8,16,32,64\}$ for all strategies and additionally $256$ and $1000$ for Vanilla UCT) was used per iteration-environment pair. One of our methods, Global Std, performs best overall.
• Figure 3: The best 6 normalized pairings score for all considered $\lambda$-strategy and exploration constant $C$ pairs as well the best UCT parameter. The score was constructed by considering all iteration budgets and all environments. One of our methods, Global Std with $C=2$, performs best overall, while standard UCT is far worse than any of these methods, proving that scale-invariant methods are necessary. The score for all $\lambda$-strategy-$C$-pairs is found in the last column of Tab. \ref{['tab:scores_strategy_and_c']} in the supplementary materials.
• Figure 5: The normalized pairings score for all considered $\lambda$-strategies. The score was constructed by considering all iteration budgets and all two-player games. Per pairing, the maximum performance over all $C$ values (i.e. $\{0.125,0.25,0.5,1,2,4,8,16,32,64\}$ for all strategies and additionally $256$ and $1000$ for Vanilla UCT) was used per iteration-environment pair. One of our methods, Global Std, performs best overall.
• Figure 6: The best 6 normalized pairings score for all considered $\lambda$-strategy and exploration constant $C$ pairs. The score was constructed by considering all iteration budgets and all two-player games. One of our methods, Global Std with $C=1$, performs best overall, showing that Global Std even yields a small performance boost over Vanilla UCT in a constant reward scale setting.

Theorems & Definitions (1)

• Definition 3.1