Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RUMAD: Reinforcement-Unifying Multi-Agent Debate

Chao Wang, Han Lin, Huaze Tang, Huijing Lin, Wenbo Ding

TL;DR

RUMAD (Reinforcement-Unifying Multi-Agent Debate), a novel framework that formulates dynamic communication topology control in MAD as a reinforcement learning (RL) problem, is presented, establishing RUMAD as a efficient and robust approach for deploying multi-agent reasoning application with practical resource constraints.

Abstract

Multi-agent debate (MAD) systems leverage collective intelligence to enhance reasoning capabilities, yet existing approaches struggle to simultaneously optimize accuracy, consensus formation, and computational efficiency. Static topology methods lack adaptability to task complexity variations, while external LLM-based coordination risks introducing privileged knowledge that compromises debate neutrality. This work presents RUMAD (Reinforcement-Unifying Multi-Agent Debate), a novel framework that formulates dynamic communication topology control in MAD as a reinforcement learning (RL) problem. RUMAD employs a content-agnostic observation scheme that captures high-level debate dynamics avoiding access to raw agent reasoning content. RUMAD uses a multi-objective reward to model solution quality, cohesion and efficiency. A PPO-trained controller dynamically adjusts edge weights in the communication graph, while a dual-threshold mechanism enables fine-grained control over both agent activation and information visibility. Experimental evaluation across MMLU, GSM8K, and GPQA benchmarks demonstrates that RUMAD achieves substantial efficiency gains, reducing token costs by over 80\%, while still improving reasoning accuracy compared to single LLM model and multiple MAD baselines. Notably, RUMAD trained exclusively on MMLU exhibits robust zero-shot generalization to out-of-domain (OOD) tasks, indicating that the learned communication strategies capture task-independent principles of effective multi-agent coordination. These results establish RUMAD as a efficient and robust approach for deploying multi-agent reasoning application with practical resource constraints.

RUMAD: Reinforcement-Unifying Multi-Agent Debate

TL;DR

RUMAD (Reinforcement-Unifying Multi-Agent Debate), a novel framework that formulates dynamic communication topology control in MAD as a reinforcement learning (RL) problem, is presented, establishing RUMAD as a efficient and robust approach for deploying multi-agent reasoning application with practical resource constraints.

Abstract

Multi-agent debate (MAD) systems leverage collective intelligence to enhance reasoning capabilities, yet existing approaches struggle to simultaneously optimize accuracy, consensus formation, and computational efficiency. Static topology methods lack adaptability to task complexity variations, while external LLM-based coordination risks introducing privileged knowledge that compromises debate neutrality. This work presents RUMAD (Reinforcement-Unifying Multi-Agent Debate), a novel framework that formulates dynamic communication topology control in MAD as a reinforcement learning (RL) problem. RUMAD employs a content-agnostic observation scheme that captures high-level debate dynamics avoiding access to raw agent reasoning content. RUMAD uses a multi-objective reward to model solution quality, cohesion and efficiency. A PPO-trained controller dynamically adjusts edge weights in the communication graph, while a dual-threshold mechanism enables fine-grained control over both agent activation and information visibility. Experimental evaluation across MMLU, GSM8K, and GPQA benchmarks demonstrates that RUMAD achieves substantial efficiency gains, reducing token costs by over 80\%, while still improving reasoning accuracy compared to single LLM model and multiple MAD baselines. Notably, RUMAD trained exclusively on MMLU exhibits robust zero-shot generalization to out-of-domain (OOD) tasks, indicating that the learned communication strategies capture task-independent principles of effective multi-agent coordination. These results establish RUMAD as a efficient and robust approach for deploying multi-agent reasoning application with practical resource constraints.
Paper Structure (33 sections, 11 equations, 3 figures, 8 tables)

This paper contains 33 sections, 11 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Multi-Agent Debate
  4. Static Workflow Generation and Dynamic Topology Control
  5. Markov Decision Process
  6. Proximal Policy Optimization
  7. Method
  8. Framework Pipeline
  9. Stage 1: Independent Initialization
  10. Stage 2: Adaptive Sparse Debate
  11. Stage 3: Consensus Formation
  12. Content-Agnostic Observation
  13. Action Space and Edge Weight Distribution Modeling
  14. Communication Budget Constraint
  15. Multi-Objective Reward
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Comparison of accuracy and average token cost. Accuracy for MMLU ($\bullet$) and GSM8K ($\blacksquare$) corresponds to the left Y-axis, while GPQA ($\blacktriangle$) corresponds to the right Y-axis. Points closer to the top-left corner represent a superior accuracy-efficiency trade-off. The results show that RUMAD variants consistently outperform baselines by achieving higher accuracy at a lower token cost.
  • Figure 2: The process pipeline of RUMAD. In the first stage, all agents gives the initial response. In the second stage, RUMAD organizes topology in stage 2.1 and the selected agents debate with each other in stage 2.2. In the last stage, the final decision is obtained via majority voting.
  • Figure 3: An example debate controlled by RUMAD.