Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic
Shuo Liu, Tianle Chen, Ryan Amiri, Christopher Amato
TL;DR
This work tackles the challenge of coordinating multiple LLM agents in a decentralized setting, addressing high variance in Monte Carlo fine-tuning and scalability concerns of centralized protocols. It proposes two MAAC-based approaches, CoLLM-CC with a centralized critic and CoLLM-DC with decentralized critics, and analyzes their relative advantages across short-horizon dense tasks and long-horizon sparse tasks. Across writing, coding, and Minecraft-related game tasks, CoLLM-CC consistently achieves superior performance and sample efficiency, while CoLLM-DC offers comparable results in dense settings but can struggle with convergence under sparse rewards. The study contributes MaAC-based frameworks for decentralized LLM collaboration, provides detailed algorithmic designs (including history representation and sequence-as-actions), and demonstrates practical benefits over static prompts and single-model baselines, with code released for reproducibility.
Abstract
Recent work has explored optimizing LLM collaboration through Multi-Agent Reinforcement Learning (MARL). However, most MARL fine-tuning approaches rely on predefined execution protocols, which often require centralized execution. Decentralized LLM collaboration is more appealing in practice, as agents can run inference in parallel with flexible deployments. Also, current approaches use Monte Carlo methods for fine-tuning, which suffer from high variance and thus require more samples to train effectively. Actor-critic methods are prevalent in MARL for dealing with these issues, so we developed Multi-Agent Actor-Critic (MAAC) methods to optimize decentralized LLM collaboration. In this paper, we analyze when and why these MAAC methods are beneficial. We propose 2 MAAC approaches, \textbf{CoLLM-CC} with a \textbf{C}entralized \textbf{C}ritic and \textbf{CoLLM-DC} with \textbf{D}ecentralized \textbf{C}ritics. Our experiments across writing, coding, and game-playing domains show that Monte Carlo methods and CoLLM-DC can achieve performance comparable to CoLLM-CC in short-horizon and dense-reward settings. However, they both underperform CoLLM-CC on long-horizon or sparse-reward tasks, where Monte Carlo methods require substantially more samples and CoLLM-DC struggles to converge. Our code is available at https://github.com/OpenMLRL/CoMLRL/releases/tag/v1.3.2.