Can Large Language Models Play Games? A Case Study of A Self-Play Approach
Hongyi Guo, Zhihan Liu, Yufeng Zhang, Zhaoran Wang
TL;DR
This work addresses decision-making in deterministic turn-based zero-sum games by integrating Large Language Models (LLMs) with Monte Carlo Tree Search (MCTS) self-play, without any additional training. The authors formalize the DTZG framework, introduce LLM-driven action pruning and value-function proxying within MCTS, and provide a sublinear suboptimality bound that accounts for the number of simulations and the quality of pruning/criticism. The theoretical analysis, supported by a LogSumExp-based pruning bound, establishes how estimation and pruning errors contribute to overall suboptimality, while experiments in chess puzzles, MiniGo, and full chess games show substantial performance gains over vanilla LLMs and conventional MCTS baselines. The results demonstrate the practical impact of leveraging pre-trained LLMs as planning aids to enhance strategic reasoning in complex games, with implications for broader decision-making tasks requiring strong priors without retraining.
Abstract
Large Language Models (LLMs) harness extensive data from the Internet, storing a broad spectrum of prior knowledge. While LLMs have proven beneficial as decision-making aids, their reliability is hampered by limitations in reasoning, hallucination phenomenon, and so on. On the other hand, Monte-Carlo Tree Search (MCTS) is a heuristic search algorithm that provides reliable decision-making solutions, achieved through recursive rollouts and self-play. However, the effectiveness of MCTS relies heavily on heuristic pruning and external value functions, particularly in complex decision scenarios. This work introduces an innovative approach that bolsters LLMs with MCTS self-play to efficiently resolve deterministic turn-based zero-sum games (DTZG), such as chess and go, without the need for additional training. Specifically, we utilize LLMs as both action pruners and proxies for value functions without the need for additional training. We theoretically prove that the suboptimality of the estimated value in our proposed method scales with $\tilde{\mathcal O}\Bigl(\frac{|\tilde {\mathcal A}|}{\sqrt{N}} + ε_\mathrm{pruner} + ε_\mathrm{critic}\Bigr)$, where \(N\) is the number of simulations, $|\tilde {\mathcal A}|$ is the cardinality of the pruned action space by LLM, and $ε_\mathrm{pruner}$ and $ε_\mathrm{critic}$ quantify the errors incurred by adopting LLMs as action space pruner and value function proxy, respectively. Our experiments in chess and go demonstrate the capability of our method to address challenges beyond the scope of MCTS and improve the performance of the directly application of LLMs.