Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeliberationBench: When Do More Voices Hurt? A Controlled Study of Multi-LLM Deliberation Protocols

Vaarunay Kaushal, Taranveer Singh

TL;DR

The paper investigates whether multi-LLM deliberation improves question-answering quality over strong simple baselines. It introduces DeliberationBench, a benchmark of $270$ questions evaluated across three seeds to compare three deliberation protocols against a best-single selection baseline, using GPT-4o as the judge. The results show a substantial negative finding: the best-single baseline achieves $82.5\pm3.3\%$ wins, while deliberation protocols peak at $13.8\pm2.6\%$, a $6.0\times$ gap with $p<0.01$, and incur $1.5$–$2.5\times$ higher costs, yielding a $15\times$ worse cost-quality ratio. These findings challenge the assumption that architectural complexity improves performance, suggesting practitioners should prioritize strong, simple baselines and carefully weigh the cost implications of multi-LLM deliberation.

Abstract

Multi-agent systems where Large Language Models (LLMs) deliberate to form consensus have gained significant attention, yet their practical value over simpler methods remains under-scrutinized. We introduce DELIBERATIONBENCH, a controlled benchmark evaluating three deliberation protocols against a strong baseline of selecting the best response from a pool of model outputs. Across 270 questions and three independent seeds (810 total evaluations), we find a striking negative result: the best-single baseline achieves an 82.5% +- 3.3% win rate, dramatically outperforming the best deliberation protocol(13.8% +- 2.6%). This 6.0x performance gap is statistically significant (p < 0.01) and comes at 1.5-2.5x higher computational cost. Our findings challenge assumptions that complexity enhances quality in multi-LLM systems.

DeliberationBench: When Do More Voices Hurt? A Controlled Study of Multi-LLM Deliberation Protocols

TL;DR

The paper investigates whether multi-LLM deliberation improves question-answering quality over strong simple baselines. It introduces DeliberationBench, a benchmark of questions evaluated across three seeds to compare three deliberation protocols against a best-single selection baseline, using GPT-4o as the judge. The results show a substantial negative finding: the best-single baseline achieves wins, while deliberation protocols peak at , a gap with , and incur – higher costs, yielding a worse cost-quality ratio. These findings challenge the assumption that architectural complexity improves performance, suggesting practitioners should prioritize strong, simple baselines and carefully weigh the cost implications of multi-LLM deliberation.

Abstract

Multi-agent systems where Large Language Models (LLMs) deliberate to form consensus have gained significant attention, yet their practical value over simpler methods remains under-scrutinized. We introduce DELIBERATIONBENCH, a controlled benchmark evaluating three deliberation protocols against a strong baseline of selecting the best response from a pool of model outputs. Across 270 questions and three independent seeds (810 total evaluations), we find a striking negative result: the best-single baseline achieves an 82.5% +- 3.3% win rate, dramatically outperforming the best deliberation protocol(13.8% +- 2.6%). This 6.0x performance gap is statistically significant (p < 0.01) and comes at 1.5-2.5x higher computational cost. Our findings challenge assumptions that complexity enhances quality in multi-LLM systems.
Paper Structure (27 sections, 5 figures, 4 tables)

This paper contains 27 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Agent LLM Systems
  4. LLM Evaluation
  5. Strong Baselines and Model Selection
  6. Methodology
  7. Benchmark Design
  8. Deliberation Protocols
  9. Protocol v1: Blind Ranking
  10. Protocol v2-A: Rubric-Based Scoring
  11. Protocol v2-B: Senate Debate
  12. Baseline: Best-Single Selection
  13. Evaluation Methodology
  14. Results
  15. Main Results
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Win rate comparison across protocols over 3 seeds. Error bars: ±1 std dev. Baseline outperforms all deliberation protocols ($p < 0.01$).
  • Figure 2: Win rate by question category. Baseline dominates across all categories.
  • Figure 3: Win rate by difficulty. Deliberation provides no advantage on harder questions.
  • Figure 4: Cost-quality frontier. Baseline occupies optimal position; deliberation protocols fall in inferior region.
  • Figure 5: Agreement between GPT-4o and Claude-3.5-Haiku judges. High agreement on baseline wins confirms robustness.