Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prepare Reasoning Language Models for Multi-Agent Debate with Self-Debate Reinforcement Learning

Chenxi Liu, Yanshuo Chen, Ruibo Chen, Tianyi Xiong, Tong Zheng, Heng Huang

TL;DR

Sdrl introduces a single-policy training framework to prepare reasoning LLMs for multi-agent debate by jointly optimizing initial solutions and debate-conditioned revisions under verifiable rewards. Grounded in a DCM-based theoretical view, it disentangles majority voting from private critique and demonstrates that critique-induced drift boosts both MAD performance and standalone reasoning. Empirically, Sdrl yields consistent gains across multiple backbones, math benchmarks, and debate frameworks, with frequency-based debate pairing providing the strongest improvements on difficult tasks. The work offers practical, scalable benefits for deploying robust, debate-capable LLMs in collaborative reasoning settings.

Abstract

The reasoning abilities of large language models (LLMs) have been substantially improved by reinforcement learning with verifiable rewards (RLVR). At test time, collaborative reasoning through Multi-Agent Debate (MAD) has emerged as a promising approach for enhancing LLM performance. However, current RLVR methods typically train LLMs to solve problems in isolation, without explicitly preparing them to synthesize and benefit from different rationales that arise during debate. In this work, we propose Self-Debate Reinforcement Learning (SDRL), a training framework that equips a single LLM with strong standalone problem-solving ability and the capability to learn from diverse reasoning trajectories in MAD. Given a prompt, SDRL first samples multiple candidate solutions, then constructs a debate context with diverse reasoning paths and generates second-turn responses conditioned on this context. Finally, SDRL jointly optimizes both the initial and debate-conditioned responses, yielding a model that is effective as both a standalone solver and a debate participant. Experiments across multiple base models and reasoning benchmarks show that SDRL improves overall MAD performance while simultaneously strengthening single model reasoning.

Prepare Reasoning Language Models for Multi-Agent Debate with Self-Debate Reinforcement Learning

TL;DR

Sdrl introduces a single-policy training framework to prepare reasoning LLMs for multi-agent debate by jointly optimizing initial solutions and debate-conditioned revisions under verifiable rewards. Grounded in a DCM-based theoretical view, it disentangles majority voting from private critique and demonstrates that critique-induced drift boosts both MAD performance and standalone reasoning. Empirically, Sdrl yields consistent gains across multiple backbones, math benchmarks, and debate frameworks, with frequency-based debate pairing providing the strongest improvements on difficult tasks. The work offers practical, scalable benefits for deploying robust, debate-capable LLMs in collaborative reasoning settings.

Abstract

The reasoning abilities of large language models (LLMs) have been substantially improved by reinforcement learning with verifiable rewards (RLVR). At test time, collaborative reasoning through Multi-Agent Debate (MAD) has emerged as a promising approach for enhancing LLM performance. However, current RLVR methods typically train LLMs to solve problems in isolation, without explicitly preparing them to synthesize and benefit from different rationales that arise during debate. In this work, we propose Self-Debate Reinforcement Learning (SDRL), a training framework that equips a single LLM with strong standalone problem-solving ability and the capability to learn from diverse reasoning trajectories in MAD. Given a prompt, SDRL first samples multiple candidate solutions, then constructs a debate context with diverse reasoning paths and generates second-turn responses conditioned on this context. Finally, SDRL jointly optimizes both the initial and debate-conditioned responses, yielding a model that is effective as both a standalone solver and a debate participant. Experiments across multiple base models and reasoning benchmarks show that SDRL improves overall MAD performance while simultaneously strengthening single model reasoning.
Paper Structure (61 sections, 6 theorems, 65 equations, 7 figures, 6 tables, 1 algorithm)

This paper contains 61 sections, 6 theorems, 65 equations, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning with Verifiable Rewards (RLVR).
  4. Multi-Agent Debate (MAD).
  5. Preliminaries
  6. Multi-Agent Debate.
  7. Reinforcement Learning.
  8. Theoretical Analysis
  9. Motivation.
  10. Role of majority vs. private critique.
  11. Methodology
  12. Single-agent reasoning
  13. Debate training
  14. Experiments
  15. Implementation Details
  16. ...and 46 more sections

Key Result

Theorem 4.4

Assume the mean-consistency condition $\bar{p}_{N(i),t-1}=p_{i,t-1}$ (the same condition under which standard MAD is a martingale in choi2025debate). Let $C_i:=m_\beta+w_i|N(i)|$, then

Figures (7)

  • Figure 1: Debate performance of $5$ agents in different debate rounds under the decentralized MAD setting.
  • Figure 2: Prompt for debate training.
  • Figure 3: Prompt for Multi-Agent evaluation. Three agents debate is shown for brevity.
  • Figure 4: Training dynamics of debate-related values using Sdrl.
  • Figure 5: Initial responses on MATH500 with $5$ agents.
  • ...and 2 more figures

Theorems & Definitions (16)

  • Definition 4.1: Critique-augmented belief update
  • Definition 4.2: Response generation via DCM
  • Definition 4.3: Advantage of private critique
  • Theorem 4.4: Critique induces belief drift
  • proof : Proof sketch
  • Lemma 4.5: Accumulated drift and diminishing returns
  • Proposition 4.6: Training increases critique advantage
  • Lemma 4.7: Correlation shrinks the effective ensemble size
  • Lemma 5.1
  • proof
  • ...and 6 more