MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning
Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, Joo-Kyung Kim
TL;DR
MAPoRL introduces a novel post-co-training paradigm that explicitly trains multiple LLMs to collaborate through multi-agent reinforcement learning, guided by a verifier-based reward signal. By formalizing collaboration as a multi-turn debate with strategic incentives and using a MARL objective (PPO) to optimize policies, MAPoRL demonstrates that co-trained agents can achieve higher collaboration and generalize to unseen tasks and domains. Experiments on GSM8K and ANLI show off-the-shelf LLMs struggle to improve via additional turns, whereas MAPoRL-trained agents gain performance as collaboration progresses and transfer these skills across datasets, including heterogeneous model pairs. The work also analyzes reward shaping strategies, showing that incentives like α1 and β0 meaningfully enhance collaborative behavior, while naive SFT on collaboration trajectories does not replicate MAPoRL’s benefits. Overall, MAPoRL provides a scalable framework for fostering robust collaboration among LLMs with potential implications for multi-agent AI systems and real-world problem solving.
Abstract
Leveraging multiple large language models (LLMs) to build collaborative multi-agentic workflows has demonstrated significant potential. However, most previous studies focus on prompting the out-of-the-box LLMs, relying on their innate capability for collaboration, which may not improve LLMs' performance as shown recently. In this paper, we introduce a new post-training paradigm MAPoRL (Multi-Agent Post-co-training for collaborative LLMs with Reinforcement Learning), to explicitly elicit the collaborative behaviors and further unleash the power of multi-agentic LLM frameworks. In MAPoRL, multiple LLMs first generate their own responses independently and engage in a multi-turn discussion to collaboratively improve the final answer. In the end, a MAPoRL verifier evaluates both the answer and the discussion, by assigning a score that verifies the correctness of the answer, while adding incentives to encourage corrective and persuasive discussions. The score serves as the co-training reward, and is then maximized through multi-agent RL. Unlike existing LLM post-training paradigms, MAPoRL advocates the co-training of multiple LLMs together using RL for better generalization. Accompanied by analytical insights, our experiments demonstrate that training individual LLMs alone is insufficient to induce effective collaboration. In contrast, multi-agent co-training can boost the collaboration performance across benchmarks, with generalization to unseen domains.