Meta-Thinking in LLMs via Multi-Agent Reinforcement Learning: A Survey

TL;DR

The paper addresses the problem of enabling reliable, introspective reasoning in LLMs by leveraging multi-agent reinforcement learning (MARL). It surveys single-agent and multi-agent meta-thinking techniques, RL-based methods like RLHF, and MARL architectures such as supervisor-worker hierarchies, agent debates, and theory-of-mind frameworks, offering a comprehensive taxonomy and roadmap. Key contributions include a structured overview of MARL strategies for meta-thinking, formalizations of meta-reward design and continual adaptation, and a synthesis of evaluation metrics and datasets used to assess meta-reasoning. The work highlights open challenges in scalability, energy efficiency, and ethics, and proposes future directions including neuroscience-inspired designs and symbolic-MARL hybrids to advance trustworthy, adaptive LLMs with robust self-correction capabilities.

Abstract

This survey explores the development of meta-thinking capabilities in Large Language Models (LLMs) from a Multi-Agent Reinforcement Learning (MARL) perspective. Meta-thinking self-reflection, assessment, and control of thinking processes is an important next step in enhancing LLM reliability, flexibility, and performance, particularly for complex or high-stakes tasks. The survey begins by analyzing current LLM limitations, such as hallucinations and the lack of internal self-assessment mechanisms. It then talks about newer methods, including RL from human feedback (RLHF), self-distillation, and chain-of-thought prompting, and each of their limitations. The crux of the survey is to talk about how multi-agent architectures, namely supervisor-agent hierarchies, agent debates, and theory of mind frameworks, can emulate human-like introspective behavior and enhance LLM robustness. By exploring reward mechanisms, self-play, and continuous learning methods in MARL, this survey gives a comprehensive roadmap to building introspective, adaptive, and trustworthy LLMs. Evaluation metrics, datasets, and future research avenues, including neuroscience-inspired architectures and hybrid symbolic reasoning, are also discussed.

Figures (5)

• Figure 1: Annual number of RL publications in AI conferences (2019–2024).
• Figure 2: The diagram illustrates a multi-agent system where a high-level agent breaks down tasks and communicates with low-level agents to execute them. The high-level agent predicts and adjusts strategies using ToM, while low-level agents provide feedback through task execution. Reflection and adaptation enable continuous improvement by refining strategies based on outcomes.
• Figure 3: A COT flowchart adapted from the arithmetic examples in Wei et al.wei2022chain. The model uses explicit reasoning steps, checks for errors, and revises before returning the final result.
• Figure 4: Overview of RL Techniques Enabling Meta-Thinking in Language Models
• Figure 5: Number of published papers referencing each dataset for evaluating LLM meta-reasoning.