Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Monte-Carlo Tree Search with Optimized Posterior Policies

Keith Frankston, Benjamin Howard

TL;DR

The paper introduces RMCTS, a breadth-first, recursive Monte-Carlo tree search that uses optimized posterior policies at every node instead of relying on adaptive UCB-based tree growth. Building on Regularized Posterior Policy optimization, RMCTS computes a locally optimal posterior $\bar{\pi}$ via Newton's method and propagates $Q$-values up the tree recursively, enabling large batch GPU inferences and substantial speedups over MCTS-UCB. Empirical results across Connect-4, Dots-and-Boxes, and Othello show RMCTS achieving orders-of-magnitude faster search for single root states and meaningful training-time gains for batches, with competitive quality that improves when simulations are doubled. The work also details an efficient implementation, simple illustrative examples, and future directions including adaptive RMCTS and extensions to MuZero.

Abstract

We introduce a recursive AlphaZero-style Monte--Carlo tree search algorithm, "RMCTS". The advantage of RMCTS over AlphaZero's MCTS-UCB is speed. In RMCTS, the search tree is explored in a breadth-first manner, so that network inferences naturally occur in large batches. This significantly reduces the GPU latency cost. We find that RMCTS is often more than 40 times faster than MCTS-UCB when searching a single root state, and about 3 times faster when searching a large batch of root states. The recursion in RMCTS is based on computing optimized posterior policies at each game state in the search tree, starting from the leaves and working back up to the root. Here we use the posterior policy explored in "Monte--Carlo tree search as regularized policy optimization" (Grill, et al.) Their posterior policy is the unique policy which maximizes the expected reward given estimated action rewards minus a penalty for diverging from the prior policy. The tree explored by RMCTS is not defined in an adaptive manner, as it is in MCTS-UCB. Instead, the RMCTS tree is defined by following prior network policies at each node. This is a disadvantage, but the speedup advantage is more significant, and in practice we find that RMCTS-trained networks match the quality of MCTS-UCB-trained networks in roughly one-third of the training time. We include timing and quality comparisons of RMCTS vs. MCTS-UCB for three games: Connect-4, Dots-and-Boxes, and Othello.

Accelerating Monte-Carlo Tree Search with Optimized Posterior Policies

TL;DR

The paper introduces RMCTS, a breadth-first, recursive Monte-Carlo tree search that uses optimized posterior policies at every node instead of relying on adaptive UCB-based tree growth. Building on Regularized Posterior Policy optimization, RMCTS computes a locally optimal posterior via Newton's method and propagates -values up the tree recursively, enabling large batch GPU inferences and substantial speedups over MCTS-UCB. Empirical results across Connect-4, Dots-and-Boxes, and Othello show RMCTS achieving orders-of-magnitude faster search for single root states and meaningful training-time gains for batches, with competitive quality that improves when simulations are doubled. The work also details an efficient implementation, simple illustrative examples, and future directions including adaptive RMCTS and extensions to MuZero.

Abstract

We introduce a recursive AlphaZero-style Monte--Carlo tree search algorithm, "RMCTS". The advantage of RMCTS over AlphaZero's MCTS-UCB is speed. In RMCTS, the search tree is explored in a breadth-first manner, so that network inferences naturally occur in large batches. This significantly reduces the GPU latency cost. We find that RMCTS is often more than 40 times faster than MCTS-UCB when searching a single root state, and about 3 times faster when searching a large batch of root states. The recursion in RMCTS is based on computing optimized posterior policies at each game state in the search tree, starting from the leaves and working back up to the root. Here we use the posterior policy explored in "Monte--Carlo tree search as regularized policy optimization" (Grill, et al.) Their posterior policy is the unique policy which maximizes the expected reward given estimated action rewards minus a penalty for diverging from the prior policy. The tree explored by RMCTS is not defined in an adaptive manner, as it is in MCTS-UCB. Instead, the RMCTS tree is defined by following prior network policies at each node. This is a disadvantage, but the speedup advantage is more significant, and in practice we find that RMCTS-trained networks match the quality of MCTS-UCB-trained networks in roughly one-third of the training time. We include timing and quality comparisons of RMCTS vs. MCTS-UCB for three games: Connect-4, Dots-and-Boxes, and Othello.
Paper Structure (11 sections, 10 equations, 2 figures, 7 tables, 4 algorithms)

This paper contains 11 sections, 10 equations, 2 figures, 7 tables, 4 algorithms.

Table of Contents

  1. AlphaZero's MCTS-UCB
  2. Regularized Policy Optimization
  3. RMCTS
  4. Implementing RMCTS efficiently
  5. Timings and quality comparisons
  6. Algorithm Descriptions
  7. A simple example
  8. Timings for MCTS-UCB vs RMCTS for three games
  9. Comparing the quality of RMCTS vs MCTS-UCB
  10. What next?
  11. Acknowledgements

Figures (2)

  • Figure 1: Illustration of UCB values over 200 simulations in bandit game with two slot arms.
  • Figure 2: Clearly the best action from state $s$ in this one-player game is to go right, reaching state $t$, where we should again choose to go right, obtaining the maximal reward of $2$.