MCTSr-Zero: Self-Reflective Psychological Counseling Dialogues Generation via Principles and Adaptive Exploration

TL;DR

This work tackles the problem of applying Monte Carlo Tree Search-guided large language models to open-ended, human-centric dialogues in psychological counseling. It introduces MCTSr-Zero, a principles-guided framework that uses domain alignment, Regeneration, and Meta-Prompt Adaptation to explore diverse dialogue trajectories, reinforced by Principled Self-Evaluation based on a 16-point psychological constitution. The authors create PsyEval, a multi-dimensional benchmark, and train PsyLLM on MCTSr-Zero-generated data, reporting state-of-the-art performance across 16 empathy-focused criteria. The results demonstrate improved alignment with therapeutic standards and open a path toward high-quality, human-centric AI for counseling, while acknowledging computational costs and potential evaluation biases as future considerations.

Abstract

The integration of Monte Carlo Tree Search (MCTS) with Large Language Models (LLMs) has demonstrated significant success in structured, problem-oriented tasks. However, applying these methods to open-ended dialogues, such as those in psychological counseling, presents unique challenges. Unlike tasks with objective correctness, success in therapeutic conversations depends on subjective factors like empathetic engagement, ethical adherence, and alignment with human preferences, for which strict "correctness" criteria are ill-defined. Existing result-oriented MCTS approaches can therefore produce misaligned responses. To address this, we introduce MCTSr-Zero, an MCTS framework designed for open-ended, human-centric dialogues. Its core innovation is "domain alignment", which shifts the MCTS search objective from predefined end-states towards conversational trajectories that conform to target domain principles (e.g., empathy in counseling). Furthermore, MCTSr-Zero incorporates "Regeneration" and "Meta-Prompt Adaptation" mechanisms to substantially broaden exploration by allowing the MCTS to consider fundamentally different initial dialogue strategies. We evaluate MCTSr-Zero in psychological counseling by generating multi-turn dialogue data, which is used to fine-tune an LLM, PsyLLM. We also introduce PsyEval, a benchmark for assessing multi-turn psychological counseling dialogues. Experiments demonstrate that PsyLLM achieves state-of-the-art performance on PsyEval and other relevant metrics, validating MCTSr-Zero's effectiveness in generating high-quality, principle-aligned conversational data for human-centric domains and addressing the LLM challenge of consistently adhering to complex psychological standards.

Figures (3)

• Figure 1: Iterative workflow of MCTSr-Zero: (1) Select & Expand responses using meta-prompts; (2) Principled Self-Evaluation against standards; (3) Backpropagation updating UCT & meta-prompts. Iterated $t$ times.
• Figure 2: The operational workflow of MCTSr-Zero. (1) Initialization: The meta-prompt $m_0$ is activated to generate initial responses $A_0$ for the user query $P$. (2) Select & Expand: The system uses the UCT value to either trigger Regeneration—using a candidate meta-prompt $m_{\mathrm{cand}}$ as the basis for generating a new answer node $A_{\mathrm{t+1}}$—or trigger Refinement to further improve an existing answer node. (3) Principled Self-Evaluation: The new answer is evaluated against predefined standards, producing a critique, score, and actionable suggestions. (4) Backpropagation: Evaluation results are propagated to update UCT scores and guide meta-prompt selection.
• Figure 3: PsyEval scores for gpt-4.1-mini with four different refinement methods across iterations. MCTSr-Zero demonstrates the highest performance.