MASTER: A Multi-Agent System with LLM Specialized MCTS

TL;DR

MASTER introduces a novel framework that replaces Monte Carlo simulations in MCTS with self-evaluated rewards guided by LLMs, coupled with an adaptive, task-aware multi-agent recruitment strategy. By weighting rewards with the LLM’s confidence and retaining backpropagation from terminal failures, MASTER achieves robust planning without extensive sampling. Across HotpotQA, WebShop, and MBPP, it attains state-of-the-art like performance while drastically reducing token usage compared to traditional tree-based MCTS approaches. The work demonstrates that careful reward design and regulated agent communication can extend MCTS-like planning to complex, non-objective tasks in a scalable, efficient manner.

Abstract

Large Language Models (LLM) are increasingly being explored for problem-solving tasks. However, their strategic planning capability is often viewed with skepticism. Recent studies have incorporated the Monte Carlo Tree Search (MCTS) algorithm to augment the planning capacity of LLM. Despite its potential, MCTS relies on extensive sampling simulations to approximate the true reward distribution, which leads to two primary issues. Firstly, MCTS is effective for tasks like the Game of Go, where simulation results can yield objective rewards (e.g., 1 for a win and 0 for a loss). However, for tasks such as question answering, the result of a simulation is the answer to the question, which cannot yield an objective reward without the ground truth. Secondly, obtaining statistically significant reward estimations typically requires a sample size exceeding 30 simulations, resulting in excessive token usage and time consumption. To address these challenges, we present the Multi-Agent System with Tactical Execution and Reasoning using LLM Specialized MCTS (MASTER), a novel framework that coordinates agent recruitment and communication through LLM specialized MCTS. This system autonomously adjusts the number of agents based on task complexity and ensures focused communication among them. Comprehensive experiments across various tasks demonstrate the effectiveness of our proposed framework. It achieves 76% accuracy on HotpotQA and 80% on WebShop, setting new state-of-the-art performance on these datasets.

Figures (2)

• Figure 1: Reasoning Tree of MASTER. Starting from $\text{Agent}_0$, $\text{Agent}_1$ and $\text{Agent}_2$ are created in the first expansion. Then the system first selects $\text{Agent}_1$ for expansion due to its higher UCT. Its child agent $\text{Agent}_3$ is a terminal agent that failed evaluation which triggers a backpropagation and lowers the UCT of $\text{Agent}_1$. Now $\text{Agent}_2$ has the highest UCT and is selected for next expansion. Its child agent, $\text{Agent}_6$ is a terminal agent and passes evaluation. The answer in it is the final answer.
• Figure 2: Reasoning Tree of case studies on three datasets.