Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Power Mean Estimation in Stochastic Monte-Carlo Tree_Search

Tuan Dam, Odalric-Ambrym Maillard, Emilie Kaufmann

TL;DR

The paper tackles theoretical gaps in UCT-based MCTS under stochastic dynamics by introducing Stochastic-Power-UCT, which uses a power-mean backup and a polynomial exploration bonus. It proves root-value estimates converge at rate $O(n^{-1/2})$, generalizing Fixed-Depth-MCTS to stochastic MDPs. The authors provide a rigorous non-stationary bandit analysis and demonstrate, through experiments on SyntheticTree, FrozenLake, and Taxi, that Stochastic-Power-UCT achieves favorable performance, especially with $p=2$. The work advances understanding of planning under uncertainty and offers practical algorithms with provable convergence guarantees for stochastic environments.

Abstract

Monte-Carlo Tree Search (MCTS) is a widely-used strategy for online planning that combines Monte-Carlo sampling with forward tree search. Its success relies on the Upper Confidence bound for Trees (UCT) algorithm, an extension of the UCB method for multi-arm bandits. However, the theoretical foundation of UCT is incomplete due to an error in the logarithmic bonus term for action selection, leading to the development of Fixed-Depth-MCTS with a polynomial exploration bonus to balance exploration and exploitation~\citep{shah2022journal}. Both UCT and Fixed-Depth-MCTS suffer from biased value estimation: the weighted sum underestimates the optimal value, while the maximum valuation overestimates it~\citep{coulom2006efficient}. The power mean estimator offers a balanced solution, lying between the average and maximum values. Power-UCT~\citep{dam2019generalized} incorporates this estimator for more accurate value estimates but its theoretical analysis remains incomplete. This paper introduces Stochastic-Power-UCT, an MCTS algorithm using the power mean estimator and tailored for stochastic MDPs. We analyze its polynomial convergence in estimating root node values and show that it shares the same convergence rate of $\mathcal{O}(n^{-1/2})$, with $n$ is the number of visited trajectories, as Fixed-Depth-MCTS, with the latter being a special case of the former. Our theoretical results are validated with empirical tests across various stochastic MDP environments.

Power Mean Estimation in Stochastic Monte-Carlo Tree_Search

TL;DR

The paper tackles theoretical gaps in UCT-based MCTS under stochastic dynamics by introducing Stochastic-Power-UCT, which uses a power-mean backup and a polynomial exploration bonus. It proves root-value estimates converge at rate , generalizing Fixed-Depth-MCTS to stochastic MDPs. The authors provide a rigorous non-stationary bandit analysis and demonstrate, through experiments on SyntheticTree, FrozenLake, and Taxi, that Stochastic-Power-UCT achieves favorable performance, especially with . The work advances understanding of planning under uncertainty and offers practical algorithms with provable convergence guarantees for stochastic environments.

Abstract

Monte-Carlo Tree Search (MCTS) is a widely-used strategy for online planning that combines Monte-Carlo sampling with forward tree search. Its success relies on the Upper Confidence bound for Trees (UCT) algorithm, an extension of the UCB method for multi-arm bandits. However, the theoretical foundation of UCT is incomplete due to an error in the logarithmic bonus term for action selection, leading to the development of Fixed-Depth-MCTS with a polynomial exploration bonus to balance exploration and exploitation~\citep{shah2022journal}. Both UCT and Fixed-Depth-MCTS suffer from biased value estimation: the weighted sum underestimates the optimal value, while the maximum valuation overestimates it~\citep{coulom2006efficient}. The power mean estimator offers a balanced solution, lying between the average and maximum values. Power-UCT~\citep{dam2019generalized} incorporates this estimator for more accurate value estimates but its theoretical analysis remains incomplete. This paper introduces Stochastic-Power-UCT, an MCTS algorithm using the power mean estimator and tailored for stochastic MDPs. We analyze its polynomial convergence in estimating root node values and show that it shares the same convergence rate of , with is the number of visited trajectories, as Fixed-Depth-MCTS, with the latter being a special case of the former. Our theoretical results are validated with empirical tests across various stochastic MDP environments.
Paper Structure (18 sections, 15 theorems, 124 equations, 1 figure, 3 tables)

This paper contains 18 sections, 15 theorems, 124 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Setting
  3. Markov Decision Process
  4. Monte-Carlo Tree Search
  5. Formalization
  6. Stochastic Power-UCT
  7. Generic UCT-like algorithm
  8. Stochastic Power-UCT
  9. Theoretical analysis
  10. Non-stationary power mean multi-armed bandit
  11. Monte-Carlo Tree Search
  12. Experiments
  13. Conclusion
  14. Outline
  15. Notations
  16. ...and 3 more sections

Key Result

Theorem 1

For $a \in [K]$, let $(\widehat{\mu}_{a,n})_{n\geq 1}$ be a sequence of estimators satisfying $\widehat{\mu}_{a,n}\overset{\alpha,\beta}{\underset{n \rightarrow \infty}{\longrightarrow}} \mu_a$ and let $\mu_\star = \max_{a} \{ \mu_a\}$. Assume that the arms are sampled according to the strategy equa If $\alpha\left(1 -\frac{b }{\alpha}\right) \leq b < \alpha$ then the sequence of estimators $\wide

Figures (1)

  • Figure 1: We show the convergence of the value estimate at the root node to the respective optimal in Synthetic tree environment.

Theorems & Definitions (32)

  • Remark 1
  • Definition 1
  • Theorem 1
  • Lemma 1
  • Theorem 2
  • proof
  • Theorem 3 (Convergence of Expected Payoff)
  • proof
  • Remark 2
  • Remark 3
  • ...and 22 more