Debate, Deliberate, Decide (D3): A Cost-Aware Adversarial Framework for Reliable and Interpretable LLM Evaluation
Chaithanya Bandi, Abir Harrasse
TL;DR
D3 presents a cost-aware, adversarial multi-agent framework for LLM evaluation that uses specialized Advocate, Judge, and Juror roles across two protocols: MORE for parallel one-round evaluation and SAMRE for budgeted, multi-round refinement. Theoretical analysis models the evaluation gap with a Beta-distributed variable, proving probabilistic convergence and enhanced score separation when leveraging parallel advocacy. Empirically, D3 achieves state-of-the-art agreement with human judgments across MT-Bench, AlignBench, and AUTO-J, while auditing biases and delineating a cost-accuracy frontier that supports practical deployment choices. These contributions provide a principled, scalable path toward reliable, interpretable, and cost-aware evaluation of LLMs, addressing critical gaps in prior single-judge and bias-prone methodologies.
Abstract
The evaluation of Large Language Models (LLMs) remains challenging due to inconsistency, bias, and the absence of transparent decision criteria in automated judging. We present Debate, Deliberate, Decide (D3), a cost-aware, adversarial multi-agent framework that orchestrates structured debate among role-specialized agents (advocates, a judge, and an optional jury) to produce reliable and interpretable evaluations. D3 instantiates two complementary protocols: (1) Multi-Advocate One-Round Evaluation (MORE), which elicits k parallel defenses per answer to amplify signal via diverse advocacy, and (2) Single-Advocate Multi-Round Evaluation (SAMRE) with budgeted stopping, which iteratively refines arguments under an explicit token budget and convergence checks. We develop a probabilistic model of score gaps that (i) characterizes reliability and convergence under iterative debate and (ii) explains the separation gains from parallel advocacy. Under mild assumptions, the posterior distribution of the round-r gap concentrates around the true difference and the probability of mis-ranking vanishes; moreover, aggregating across k advocates provably increases expected score separation. We complement theory with a rigorous experimental suite across MT-Bench, AlignBench, and AUTO-J, showing state-of-the-art agreement with human judgments (accuracy and Cohen's kappa), reduced positional and verbosity biases via anonymization and role diversification, and a favorable cost-accuracy frontier enabled by budgeted stopping. Ablations and qualitative analyses isolate the contributions of debate, aggregation, and anonymity. Together, these results establish D3 as a principled, practical recipe for reliable, interpretable, and cost-aware LLM evaluation.