Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provably Efficient Long-Horizon Exploration in Monte Carlo Tree Search through State Occupancy Regularization

Liam Schramm, Abdeslam Boularias

TL;DR

This work tackles the challenge of long-horizon exploration in MCTS by introducing Volume-MCTS, a tree-search algorithm that optimizes a state-occupancy regularized objective to encourage broad and efficient exploration. By proving convexity properties on trees and deriving closed-form node policies, the authors connect Voronoi bias, count-based exploration, and occupancy regularization, and they implement a KD-tree-backed Value/Policy estimation to realize scalable exploration. They provide non-asymptotic high-probability guarantees on exploration speed and demonstrate empirical improvements over AlphaZero variants, CBE-augmented methods, and SBMP in maze and quadcopter-like robotic tasks. The approach offers a principled, network-friendly way to drive long-horizon exploration in robotics and related domains, with potential extensions to stochastic dynamics and action-dependent rewards.

Abstract

Monte Carlo tree search (MCTS) has been successful in a variety of domains, but faces challenges with long-horizon exploration when compared to sampling-based motion planning algorithms like Rapidly-Exploring Random Trees. To address these limitations of MCTS, we derive a tree search algorithm based on policy optimization with state occupancy measure regularization, which we call {\it Volume-MCTS}. We show that count-based exploration and sampling-based motion planning can be derived as approximate solutions to this state occupancy measure regularized objective. We test our method on several robot navigation problems, and find that Volume-MCTS outperforms AlphaZero and displays significantly better long-horizon exploration properties.

Provably Efficient Long-Horizon Exploration in Monte Carlo Tree Search through State Occupancy Regularization

TL;DR

This work tackles the challenge of long-horizon exploration in MCTS by introducing Volume-MCTS, a tree-search algorithm that optimizes a state-occupancy regularized objective to encourage broad and efficient exploration. By proving convexity properties on trees and deriving closed-form node policies, the authors connect Voronoi bias, count-based exploration, and occupancy regularization, and they implement a KD-tree-backed Value/Policy estimation to realize scalable exploration. They provide non-asymptotic high-probability guarantees on exploration speed and demonstrate empirical improvements over AlphaZero variants, CBE-augmented methods, and SBMP in maze and quadcopter-like robotic tasks. The approach offers a principled, network-friendly way to drive long-horizon exploration in robotics and related domains, with potential extensions to stochastic dynamics and action-dependent rewards.

Abstract

Monte Carlo tree search (MCTS) has been successful in a variety of domains, but faces challenges with long-horizon exploration when compared to sampling-based motion planning algorithms like Rapidly-Exploring Random Trees. To address these limitations of MCTS, we derive a tree search algorithm based on policy optimization with state occupancy measure regularization, which we call {\it Volume-MCTS}. We show that count-based exploration and sampling-based motion planning can be derived as approximate solutions to this state occupancy measure regularized objective. We test our method on several robot navigation problems, and find that Volume-MCTS outperforms AlphaZero and displays significantly better long-horizon exploration properties.
Paper Structure (43 sections, 23 theorems, 54 equations, 3 figures, 3 tables, 4 algorithms)

This paper contains 43 sections, 23 theorems, 54 equations, 3 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Definitions
  3. Background
  4. State Space Regularization
  5. Count-Based Exploration (CBE)
  6. Sampling-based Motion Planning
  7. Monte Carlo Tree Search
  8. MCTS as Regularized Policy Optimization:
  9. RRT and Count-Based Exploration as Regularized Policy Optimization
  10. Connection to RRT
  11. Connection to Count-based Exploration
  12. Volume-MCTS Algorithm
  13. Guarantees
  14. Tree Search Algorithm
  15. Expansion, Value Estimation $\&$ K-D Tree Backprop
  16. ...and 28 more sections

Key Result

Proposition 1

Suppose $f(t) = - \ln{t}$ and $\psi(T, d^\pi)(s)$ is a partition density estimator, where the associated measure of any node $n$ is $\operatorname{Vol}(n)$. Then $\mathcal{L}$ has a unique optimizer $d^{\pi*}$, such that $d^{\pi*}(n) = \frac{\lambda}{\alpha - \mathcal{V}(n)} \operatorname{Vol}(n)$,

Figures (3)

  • Figure 1: Comparison of AlphaZero and Volume-MCTS on the geometric maze environment
  • Figure 2: Reward and success rate on Quadcopter environment
  • Figure :

Theorems & Definitions (43)

  • Definition 1
  • Definition 2
  • Proposition 1
  • Proposition 2
  • Theorem 1
  • Theorem 2
  • Definition 3
  • Theorem 3
  • Corollary 1
  • Definition 4
  • ...and 33 more