Agent-GSPO: Communication-Efficient Multi-Agent Systems via Group Sequence Policy Optimization

TL;DR

This work tackles the high token costs of free-for-all multi-agent systems by introducing Agent-GSPO, a framework that optimizes token economy using sequence-level reinforcement learning. It combines a communication-aware reward with the stable, memory-efficient Group Sequence Policy Optimization (GSPO) objective and a dual budget constraint to train agents that balance task performance with concise communication. Key contributions include a practical reward design penalizing verbosity, a sequence-level GSPO training regime, and evidence of emergent strategies like strategic silence, achieving state-of-the-art results across seven reasoning benchmarks with dramatically lower token usage. The approach offers a scalable, economically viable path toward more efficient multi-agent collaboration in large language model-based systems.

Abstract

To combat the prohibitive communication costs of ``free-for-all" multi-agent systems (MAS), we introduce \textbf{Agent-GSPO}, a framework that directly optimizes for token economy using sequence-level reinforcement learning. Agent-GSPO leverages the stable and memory-efficient Group Sequence Policy Optimization (GSPO) algorithm to train agents on a communication-aware reward that explicitly penalizes verbosity. Across seven reasoning benchmarks, Agent-GSPO not only achieves new state-of-the-art performance but does so with a fraction of the token consumption of existing methods. By fostering emergent strategies like ``strategic silence," our approach provides a practical blueprint for developing scalable and economically viable multi-agent systems.

Figures (2)

• Figure 1: Overview of the Agent-GSPO training pipeline. The Actor Network ($\pi_{\theta_{\text{old}}}$) samples a group of candidate responses. Each response receives a communication-aware reward ($r$), which balances task success ($r_{\text{task}}$) against a token-length penalty. The GSPO module processes these sequence-level rewards to update the Actor's policy ($\pi_\theta$), training the agent to favor concise and effective communication.
• Figure 2: Ablation study on the MMLU benchmark.