Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Anytime Sequential Halving in Monte-Carlo Tree Search

Dominic Sagers, Mark H. M. Winands, Dennis J. N. J. Soemers

TL;DR

This paper proposes an anytime version of the MCTS algorithm, which can be halted at any arbitrary time and still return a satisfactory result, while being designed such that it approximates the behavior of Sequential Halving.

Abstract

Monte-Carlo Tree Search (MCTS) typically uses multi-armed bandit (MAB) strategies designed to minimize cumulative regret, such as UCB1, as its selection strategy. However, in the root node of the search tree, it is more sensible to minimize simple regret. Previous work has proposed using Sequential Halving as selection strategy in the root node, as, in theory, it performs better with respect to simple regret. However, Sequential Halving requires a budget of iterations to be predetermined, which is often impractical. This paper proposes an anytime version of the algorithm, which can be halted at any arbitrary time and still return a satisfactory result, while being designed such that it approximates the behavior of Sequential Halving. Empirical results in synthetic MAB problems and ten different board games demonstrate that the algorithm's performance is competitive with Sequential Halving and UCB1 (and their analogues in MCTS).

Anytime Sequential Halving in Monte-Carlo Tree Search

TL;DR

This paper proposes an anytime version of the MCTS algorithm, which can be halted at any arbitrary time and still return a satisfactory result, while being designed such that it approximates the behavior of Sequential Halving.

Abstract

Monte-Carlo Tree Search (MCTS) typically uses multi-armed bandit (MAB) strategies designed to minimize cumulative regret, such as UCB1, as its selection strategy. However, in the root node of the search tree, it is more sensible to minimize simple regret. Previous work has proposed using Sequential Halving as selection strategy in the root node, as, in theory, it performs better with respect to simple regret. However, Sequential Halving requires a budget of iterations to be predetermined, which is often impractical. This paper proposes an anytime version of the algorithm, which can be halted at any arbitrary time and still return a satisfactory result, while being designed such that it approximates the behavior of Sequential Halving. Empirical results in synthetic MAB problems and ten different board games demonstrate that the algorithm's performance is competitive with Sequential Halving and UCB1 (and their analogues in MCTS).

Paper Structure

This paper contains 9 sections, 4 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Time-Based and Anytime Sequential Halving
  4. Experiments
  5. Synthetic Multi-Armed Bandit Problems
  6. Board Games
  7. Discussion
  8. Conclusion and Future Work

Figures (4)

  • Figure 1: Example of one complete run of Sequential Halving with 15,000 iterations. Only the root node and its immediate children are depicted. Circles in the same vertical bar depict the same node, in different rounds of the algorithm. The best half of nodes in each round are colored. Numbers inside the nodes denote the total number of iterations allocated to that node up to and including the corresponding round.
  • Figure 2: Visualization of two passes made by Anytime Sequential Halving. Only the root node and its immediate children are depicted. Circles in the same vertical bar depict the same node, in different rounds of the algorithm. The best half of nodes in each round of each pass are colored. Numbers inside the nodes denote the total number of iterations allocated to that node up to and including the corresponding pass and round.
  • Figure 3: Simple regret, averaged over 100 MAB problems, for four MAB algorithms.
  • Figure 4: Win percentages averaged over all ten games against UCT (left) and H-MCTS (right). $95\%$ confidence intervals are too small to be visible.