Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Efficient Algorithm for Thresholding Monte Carlo Tree Search

Shoma Nameki, Atsuyoshi Nakamura, Junpei Komiyama, Koji Tabata

TL;DR

This work tackles Thresholding Monte Carlo Tree Search under a fixed confidence regime, where the goal is to determine if the root value exceeds a threshold $\theta$ in a MAX/MIN tree with leaf rewards from an exponential family. It develops RD‑Tracking‑TMCTS, a ratio‑based variant of Track‑and‑Stop that uses GLR stopping to achieve asymptotically optimal sample complexity, quantified by $1/d_{s_0}(\bm{\mu})$, while enabling $O(D \log K)$ time per step on balanced trees. The authors prove almost‑sure and expected stopping‑time bounds matching the lower bound, and demonstrate substantial empirical gains over baselines in synthetic tree experiments. They also adapt the framework to Good Action Identification, preserving the same asymptotic optimality properties and extending the utility to action identification in MCTS settings.

Abstract

We introduce the Thresholding Monte Carlo Tree Search problem, in which, given a tree $\mathcal{T}$ and a threshold $θ$, a player must answer whether the root node value of $\mathcal{T}$ is at least $θ$ or not. In the given tree, `MAX' or `MIN' is labeled on each internal node, and the value of a `MAX'-labeled (`MIN'-labeled) internal node is the maximum (minimum) of its child values. The value of a leaf node is the mean reward of an unknown distribution, from which the player can sample rewards. For this problem, we develop a $δ$-correct sequential sampling algorithm based on the Track-and-Stop strategy that has asymptotically optimal sample complexity. We show that a ratio-based modification of the D-Tracking arm-pulling strategy leads to a substantial improvement in empirical sample complexity, as well as reducing the per-round computational cost from linear to logarithmic in the number of arms.

An Efficient Algorithm for Thresholding Monte Carlo Tree Search

TL;DR

This work tackles Thresholding Monte Carlo Tree Search under a fixed confidence regime, where the goal is to determine if the root value exceeds a threshold in a MAX/MIN tree with leaf rewards from an exponential family. It develops RD‑Tracking‑TMCTS, a ratio‑based variant of Track‑and‑Stop that uses GLR stopping to achieve asymptotically optimal sample complexity, quantified by , while enabling time per step on balanced trees. The authors prove almost‑sure and expected stopping‑time bounds matching the lower bound, and demonstrate substantial empirical gains over baselines in synthetic tree experiments. They also adapt the framework to Good Action Identification, preserving the same asymptotic optimality properties and extending the utility to action identification in MCTS settings.

Abstract

We introduce the Thresholding Monte Carlo Tree Search problem, in which, given a tree and a threshold , a player must answer whether the root node value of is at least or not. In the given tree, `MAX' or `MIN' is labeled on each internal node, and the value of a `MAX'-labeled (`MIN'-labeled) internal node is the maximum (minimum) of its child values. The value of a leaf node is the mean reward of an unknown distribution, from which the player can sample rewards. For this problem, we develop a -correct sequential sampling algorithm based on the Track-and-Stop strategy that has asymptotically optimal sample complexity. We show that a ratio-based modification of the D-Tracking arm-pulling strategy leads to a substantial improvement in empirical sample complexity, as well as reducing the per-round computational cost from linear to logarithmic in the number of arms.
Paper Structure (31 sections, 30 theorems, 186 equations, 3 figures, 10 algorithms)

This paper contains 31 sections, 30 theorems, 186 equations, 3 figures, 10 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setting
  3. Optimal Allocation
  4. Track-and-Stop Algorithms
  5. Stopping Rule
  6. Sampling Rules
  7. Advantages of the Ratio-based Sampling
  8. Algorithms
  9. Analysis
  10. Assumptions on the True Parameters
  11. Almost-Sure Upper Bound of Sample Complexity
  12. Upper Bound of Expected Sample Complexity
  13. Time Complexity
  14. Experimental Results
  15. Good Action Identification Problem
  16. ...and 16 more sections

Key Result

Theorem 3.1

The following inequality holds: where $\Delta$ is the simplex over leaves

Figures (3)

  • Figure 1: A six-armed example to demonstrate the inefficiency of the original D-tracking.
  • Figure 2: Recursive calculation of $I_s(t)$
  • Figure 3: Ratio of the average stopping time to the lower bound for different values of $\delta$. Shaded regions indicate the standard deviation.

Theorems & Definitions (55)

  • Definition 2.1: $\delta$-correctness
  • Theorem 3.1: pmlr-v49-garivier16aDBLP:conf/nips/DegenneK19
  • Theorem 3.2
  • Theorem 4.1
  • Corollary 4.2: Corollary of Proposition 15 in kaufmann2021
  • Example 4.3
  • Lemma 4.4
  • Theorem 5.3
  • Lemma 5.4
  • Corollary 5.5
  • ...and 45 more