On the Convergence Rate of MCTS for the Optimal Value Estimation in Markov Decision Processes
Hyeong Soo Chang
TL;DR
This work analyzes the convergence rates of MCTS-based value estimation in finite-horizon MDPs, showing that a UCB1-based framing yields an asymptotically faster rate of $O(\ln n / n)$ than the traditional UCT/UCT-C approach, which attains $O(1/\sqrt{n})$ under certain deterministic assumptions. It argues that UCT-based methods incur time and space costs tied to the state space, challenging their intended scalability and theoretical grounding. The study distinguishes deterministic and stochastic MDP settings, deriving a $O( H|A| / \min_h(\Delta^{h}_{\min})^2 \cdot 1/\sqrt{n} )$ bound for deterministic MDPs with UCT-C and showing stochastic cases require reductions to deterministic subproblems, introducing large constants and state-space dependence. Overall, the paper highlights the lack of general theoretical guarantees for UCT-based MCTS in stochastic domains and suggests that, within the proposed framework, the best asymptotic rate is $O(\ln n / n)$, motivating future work on variants that combine DP principles with improved concentration properties.
Abstract
A recent theoretical analysis of a Monte-Carlo tree search (MCTS) method properly modified from the ``upper confidence bound applied to trees" (UCT) algorithm established a surprising result, due to a great deal of empirical successes reported from heuristic usage of UCT with relevant adjustments for various problem domains in the literature, that its rate of convergence of the expected absolute error to zero is $O(1/\sqrt{n})$ in estimating the optimal value at an initial state in a finite-horizon Markov decision process (MDP), where $n$ is the number of simulations. We strengthen this dispiriting slow convergence result by arguing within a simpler algorithmic framework in the perspective of MDP, apart from the usual MCTS description, that the simpler strategy, called ``upper confidence bound 1" (UCB1) for multi-armed bandit problems, when employed as an instance of MCTS by setting UCB1's arm set to be the policy set of the underlying MDP, has an asymptotically faster convergence-rate of $O(\ln n / n)$. We also point out that the UCT-based MCTS in general has the time and space complexities that depend on the size of the state space in the worst case, which contradicts the original design spirit of MCTS. Unless heuristically used, UCT-based MCTS has yet to have theoretical supports for its applicabilities.