Combining LLMs with Logic-Based Framework to Explain MCTS

TL;DR

This work tackles the transparency gap in sequential planning by introducing a logic-guided, LLM-based explanation framework for $MCTS$ within a $MDP$ setting. By converting user queries into logic statements and leveraging a three-layer evidence system, CTL model checking, and retrieval-augmented knowledge, the approach ensures explanations are factually aligned with environmental dynamics. The framework supports flexible, interactive querying (post-hoc and background knowledge) and demonstrates substantial improvements in factual consistency over baseline LLMs on a paratransit planning testbed. The results suggest practical relevance for deploying interpretable planning algorithms in real-world domains where trust and accountability are critical.

Abstract

In response to the lack of trust in Artificial Intelligence (AI) for sequential planning, we design a Computational Tree Logic-guided large language model (LLM)-based natural language explanation framework designed for the Monte Carlo Tree Search (MCTS) algorithm. MCTS is often considered challenging to interpret due to the complexity of its search trees, but our framework is flexible enough to handle a wide range of free-form post-hoc queries and knowledge-based inquiries centered around MCTS and the Markov Decision Process (MDP) of the application domain. By transforming user queries into logic and variable statements, our framework ensures that the evidence obtained from the search tree remains factually consistent with the underlying environmental dynamics and any constraints in the actual stochastic control process. We evaluate the framework rigorously through quantitative assessments, where it demonstrates strong performance in terms of accuracy and factual consistency.

Figures (1)

• Figure 1: We explain sequential planning by combining domain knowledge, search process, and logical reasoning.