Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Monte Carlo Search Algorithms Discovering Monte Carlo Tree Search Exploration Terms

Tristan Cazenave

TL;DR

The paper tackles improving Monte Carlo Tree Search by discovering new root exploration terms through an empirical, expression-based search. It introduces an expression discovery game and an AMAF-based sampling approach to rapidly evaluate candidate terms, aiming to enhance PUCT and SHUSS, especially under small budgeting. The authors demonstrate that the discovered terms can achieve competitive performance against standard PUCT and improve SHUSS, supported by a Go-focused experimental pipeline including a Katago Go dataset and a fast SHUSS evaluation dataset. The work offers a simple, scalable method for AI-driven algorithm design with potential applicability beyond Go to other decision problems and domains.

Abstract

Monte Carlo Tree Search and Monte Carlo Search have good results for many combinatorial problems. In this paper we propose to use Monte Carlo Search to design mathematical expressions that are used as exploration terms for Monte Carlo Tree Search algorithms. The optimized Monte Carlo Tree Search algorithms are PUCT and SHUSS. We automatically design the PUCT and the SHUSS root exploration terms. For small search budgets of 32 evaluations the discovered root exploration terms make both algorithms competitive with usual PUCT.

Monte Carlo Search Algorithms Discovering Monte Carlo Tree Search Exploration Terms

TL;DR

The paper tackles improving Monte Carlo Tree Search by discovering new root exploration terms through an empirical, expression-based search. It introduces an expression discovery game and an AMAF-based sampling approach to rapidly evaluate candidate terms, aiming to enhance PUCT and SHUSS, especially under small budgeting. The authors demonstrate that the discovered terms can achieve competitive performance against standard PUCT and improve SHUSS, supported by a Go-focused experimental pipeline including a Katago Go dataset and a fast SHUSS evaluation dataset. The work offers a simple, scalable method for AI-driven algorithm design with potential applicability beyond Go to other decision problems and domains.

Abstract

Monte Carlo Tree Search and Monte Carlo Search have good results for many combinatorial problems. In this paper we propose to use Monte Carlo Search to design mathematical expressions that are used as exploration terms for Monte Carlo Tree Search algorithms. The optimized Monte Carlo Tree Search algorithms are PUCT and SHUSS. We automatically design the PUCT and the SHUSS root exploration terms. For small search budgets of 32 evaluations the discovered root exploration terms make both algorithms competitive with usual PUCT.
Paper Structure (22 sections, 12 equations, 5 figures, 8 tables, 2 algorithms)

This paper contains 22 sections, 12 equations, 5 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Monte Carlo Tree Search
  3. PUCT
  4. The AMAF prior
  5. Sampling
  6. Sequential Halving
  7. Sequential Halving Using Scores
  8. Using an exploration term for the selection of moves in SHUSS
  9. Using an exploration term for the selection of moves in PUCT
  10. Generating Mathematical Expressions
  11. The expression discovery game
  12. Sampling
  13. Experimental Results
  14. The computer Go dataset
  15. The neural network
  16. ...and 7 more sections

Figures (5)

  • Figure 1: The three steps of MCTS. The first step is the tree descent using the exploration term to choose among the children. The second step is adding a new leaf associated to an evaluation of the state by the value network. The third step is updating the statistics in the tree with the evaluation.
  • Figure 2: Evolution of the Accuracy.
  • Figure 3: Evolution of the MSE.
  • Figure 4: Evolution of the MAE.
  • Figure 5: Evolution of the best expression accuracy with the logarithm of the sampling search time with doubling search times. Each measure is the average of 100 runs of the sampling algorithms. Using the AMAF prior improves the results. It finds the same accuracy more than 8 times faster than the uniform sampling algorithm. The temperature of the AMAF sampling is set to 5. The dataset used is the Sequential Halving moves with 128 evaluations dataset and the exploration terms are scored using 32 evaluations on each state out of the 2,000 states.