Towards Scalable Oversight with Collaborative Multi-Agent Debate in Error Detection

TL;DR

The paper tackles scalable oversight by focusing on reliable error detection in LLM outputs, identifying debate hacking as a major weakness of zero-sum MAD. It introduces Collaborative Multi-Agent Debate (ColMAD), a non-zero-sum framework in which debaters collaboratively surface missing information through evidence verification, self-auditing, and confidence calibration. The authors establish theoretical guarantees that collaboration can outperform competitive MAD and single-agent baselines when new information is provided, and they validate ColMAD experimentally on the ReaLMistake benchmark, showing consistent gains (up to ~4% over single agents) and robustness across LLM pairings while CopMAD often degrades performance. Collectively, the work offers a practical, scalable approach to error detection in LLMs with more human-aligned explanations, contributing to safer supervision of increasingly capable AI systems.

Abstract

Accurate detection of errors in large language models (LLM) responses is central to the success of scalable oversight, or providing effective supervision to superhuman intelligence. Yet, self-diagnosis is often unreliable on complex tasks unless aided by reliable external feedback. Multi-agent debate (MAD) seems to be a natural alternative to external feedback: multiple LLMs provide complementary perspectives and cross-checks for error detection. However, prior MAD protocols frame debate as a zero-sum game, where the debaters compete to win the game instead of seeking the truth. Consequently, it leads to debate hacking: debaters tend to mislead the judge by misinterpreting the task or presenting overconfident claims, which introduce more mistakes and underperform single-agent methods. To mitigate the issue, we introduce a new collaborative MAD protocol, termed ColMAD, that reframes MAD as a non-zero sum game. Specifically, ColMAD encourages multiple agents to criticize each other in a supportive way, such that they can complement the missing points of each other. Therefore, the judge agent can make a more informative conclusion based on more comprehensive evidence. Empirically, we show that ColMAD significantly outperforms previous competitive MAD by 19% and brings non-trivial improvements over single-agent methods in error detection.

Figures (8)

• Figure 1: Comparison between competitive multi-agent debate (CopMAD) and collaborative multi-agent debate (ColMAD). Given an LLM response to a task of generating a math problem according to some given requirements, it is required to examine whether the LLM response meets all the requirements properly. The original MAD scheme suffers from debate hacking. Due to the zero-sum nature, the dishonest debaters tend to mislead the judge with misinterpreted pieces of evidence ("including the cost of the scooter is unnecessary") or overconfident claims ("fundamentally flawed"). Instead, collaborative debate aims to complement the missing information of each other in order to assist the judge in making a more informed decision.
• Figure 2: Error reductions of prevalent LLMs in detecting errors of GPT-4. The numbers refer to the reduced errors following the oracle collaboration via Eq. (\ref{['eq:oracle_collab']}). It can be found that different LLMs are less likely to make mistakes simultaneously. When incorporating LLMs with higher heterogeneity, such as those from different companies, the error reduction rates will be higher.
• Figure 3: Pitfalls of previous MAD protocols. Under previous MAD schemes, such as CopMAD or SoM, the debate results are lower than any of the LLMs involved in the debate in most cases. In contrast, ColMAD significantly improves CopMAD and outperforms the use of a single LLM.
• Figure 4: Illustration of the debate hacking issue in CopMAD. We observe three typical debate hacking behaviors: (i) Fake evidence that dishonest debaters misinterpret the requirements of the task; (ii) Overconfident claims that dishonest debaters use an overconfident tone to mislead the judge; and (iii) Fallacious argument that dishonest debaters turn the focus to side and meaningless points.
• Figure 5: (a) shows the results in F2 scores of ColMAD performance under different combinations of LLMs, where the diagonal line shows the single-agent performance. (b) shows the rate of alignment to the ground-truth explanations given by CopMAD and ColMAD, where "ML-" refers to the combination of GPT4o-mini and Llama3.1-70B, and "LD-" refers to the combination of Llama3.1-70B and DeepSeek-R1. ColMAD yields more reasonable explanations than CopMAD.

Theorems & Definitions (6)

• Proposition 2.2: Pitfalls of dishonest competitive debating
• Proposition 2.3: Collaborative debating
• Proposition A.2: Restatement of Proposition \ref{['thm:pitfal_debating']}
• proof
• Proposition A.3: Restatement of Proposition \ref{['thm:comad']}
• proof