LLM Collaboration With Multi-Agent Reinforcement Learning
Shuo Liu, Tianle Chen, Zeyu Liang, Xueguang Lyu, Christopher Amato
TL;DR
The paper models LLM collaboration as a cooperative MARL problem under a Dec-POMDP, addressing the lack of coordinated fine-tuning by introducing MAGRPO. MAGRPO uses group-relative advantages and centralized training with decentralized execution, aided by group Monte Carlo estimates to stabilize learning. Empirical results on writing and coding tasks show MAGRPO improves both the efficiency and quality of multi-agent outputs, and reveal emergent cooperation schemes. The work highlights challenges and points to future MARL-based methods for scalable, robust LLM collaboration with open-source tooling.
Abstract
A large amount of work has been done in Multi-Agent Systems (MAS) for modeling and solving problems with multiple interacting agents. However, most LLMs are pretrained independently and not specifically optimized for coordination. Existing LLM fine-tuning frameworks rely on individual rewards, which require complex reward designs for each agent to encourage collaboration. To address these challenges, we model LLM collaboration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. We develop a multi-agent, multi-turn algorithm, Multi-Agent Group Relative Policy Optimization (MAGRPO), to solve it, building on current RL approaches for LLMs as well as MARL techniques. Our experiments on LLM writing and coding collaboration demonstrate that fine-tuning MAS with MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation. Our approach opens the door to using other MARL methods for LLMs and highlights the associated challenges. Our code is available at https://github.com/OpenMLRL/CoMLRL.