Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learning

TL;DR

<3-5 sentence high-level summary> This work addresses the rigidity of fixed collaboration protocols in multi-agent LLM reasoning by proposing the Meta-Policy Deliberation Framework (MPDF), where agents learn decentralized meta-policies over high-level deliberation actions (Persist, Refine, Concede) based on internal cognitive states. To stabilize learning in sparse, heterogeneous feedback, it introduces SoftRankPO, a rank-based, KL-regularized policy optimizer that reduces sensitivity to reward scale. A two-stage training pipeline combines supervised bootstrapping with MARL fine-tuning and a consensus-driven reward shaping scheme to align individual improvements with collective success. Empirically, MPDF with SoftRankPO achieves strong gains across six benchmarks and demonstrates robust performance across diverse backbones, along with lower token costs. This work advances adaptive, meta-cognitive coordination in multi-agent LLM systems and provides a scalable approach to learn dynamic, deliberative strategies.

Abstract

Multi-agent systems of large language models (LLMs) show promise for complex reasoning, but their effectiveness is often limited by fixed collaboration protocols. These frameworks typically focus on macro-level orchestration while overlooking agents' internal deliberative capabilities. This critical meta-cognitive blindspot treats agents as passive executors unable to adapt their strategy based on internal cognitive states like uncertainty or confidence. We introduce the Meta-Policy Deliberation Framework (MPDF), where agents learn a decentralized policy over a set of high-level meta-cognitive actions: Persist, Refine, and Concede. To overcome the instability of traditional policy gradients in this setting, we develop SoftRankPO, a novel reinforcement learning algorithm. SoftRankPO stabilizes training by shaping advantages based on the rank of rewards mapped through smooth normal quantiles, making the learning process robust to reward variance. Experiments show that MPDF with SoftRankPO achieves a a 4-5% absolute gain in average accuracy across five mathematical and general reasoning benchmarks compared to six state-of-the-art heuristic and learning-based multi-agent reasoning algorithms. Our work presents a paradigm for learning adaptive, meta-cognitive policies for multi-agent LLM systems, shifting the focus from designing fixed protocols to learning dynamic, deliberative strategies.

Figures (7)

• Figure 1: Evolving from fixed static debate to trainable meta-policy deliberation, with dynamic strategy adaptation for multi-agent collaboration.
• Figure 2: An overview of the Meta-Policy Deliberation Framework (MPDF). (a) Meta-Cognitive State Representation: Each agent constructs a cognitive state with decision schema, reasoning profile and introspective confidence. (b) Meta-Policy Deliberation: The agent's meta-policy network selects one of three high-level actions: Persist (if confident), Refine (to improve its own solution), or Concede (to yield to a peer solution). (c) Environment Interaction: The chosen action leads to a new response, yielding a reward and transitioning the system to the next state. (d) SoftRankPO: a stable policy optimization RL algorithm.
• Figure 3: Accuracy curves during deliberative training on GSM8K (top) and MATH (bottom) using the LLaMA3-8B backbone. Our method consistently achieves stable and faster convergence across both benchmarks.
• Figure 4: Our framework scales well with more rounds and agents, with diminishing returns beyond optimal settings.
• Figure 5: Stability under different exponential reward scales. SoftRankPO remains robustness across multiple scales.