Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DEBATE: Devil's Advocate-Based Assessment and Text Evaluation

Alex Kim, Keonwoo Kim, Sangwon Yoon

TL;DR

DEBATE tackles the bias and stability limitations of single-agent LLM evaluators by introducing a three-agent, Devil's Advocate–powered framework for NLG evaluation. Through iterative debates among Commander, Scorer, and Critic, and with a Critic that actively challenges scores, DEBATE achieves superior correlations with human judgments on SummEval and Topical-Chat, outperforming prior baselines. The approach demonstrates robustness across LLM families and reveals that debate depth, critic persona, and tie-breaker strategies meaningfully affect performance. While offering improved alignment with human ratings, the method also incurs higher computational costs and depends on the capabilities of the underlying LLMs, suggesting directions for future efficiency and generalization.

Abstract

As natural language generation (NLG) models have become prevalent, systematically assessing the quality of machine-generated texts has become increasingly important. Recent studies introduce LLM-based evaluators that operate as reference-free metrics, demonstrating their capability to adeptly handle novel tasks. However, these models generally rely on a single-agent approach, which, we argue, introduces an inherent limit to their performance. This is because there exist biases in LLM agent's responses, including preferences for certain text structure or content. In this work, we propose DEBATE, an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil's Advocate. Within the framework, one agent is instructed to criticize other agents' arguments, potentially resolving the bias in LLM agent's answers. DEBATE substantially outperforms the previous state-of-the-art methods in two meta-evaluation benchmarks in NLG evaluation, SummEval and TopicalChat. We also show that the extensiveness of debates among agents and the persona of an agent can influence the performance of evaluators.

DEBATE: Devil's Advocate-Based Assessment and Text Evaluation

TL;DR

DEBATE tackles the bias and stability limitations of single-agent LLM evaluators by introducing a three-agent, Devil's Advocate–powered framework for NLG evaluation. Through iterative debates among Commander, Scorer, and Critic, and with a Critic that actively challenges scores, DEBATE achieves superior correlations with human judgments on SummEval and Topical-Chat, outperforming prior baselines. The approach demonstrates robustness across LLM families and reveals that debate depth, critic persona, and tie-breaker strategies meaningfully affect performance. While offering improved alignment with human ratings, the method also incurs higher computational costs and depends on the capabilities of the underlying LLMs, suggesting directions for future efficiency and generalization.

Abstract

As natural language generation (NLG) models have become prevalent, systematically assessing the quality of machine-generated texts has become increasingly important. Recent studies introduce LLM-based evaluators that operate as reference-free metrics, demonstrating their capability to adeptly handle novel tasks. However, these models generally rely on a single-agent approach, which, we argue, introduces an inherent limit to their performance. This is because there exist biases in LLM agent's responses, including preferences for certain text structure or content. In this work, we propose DEBATE, an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil's Advocate. Within the framework, one agent is instructed to criticize other agents' arguments, potentially resolving the bias in LLM agent's answers. DEBATE substantially outperforms the previous state-of-the-art methods in two meta-evaluation benchmarks in NLG evaluation, SummEval and TopicalChat. We also show that the extensiveness of debates among agents and the persona of an agent can influence the performance of evaluators.
Paper Structure (51 sections, 2 figures, 3 tables, 1 algorithm)

This paper contains 51 sections, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. Overview
  4. Predefined template
  5. Multi-agent scoring framework
  6. Devil's Advocate
  7. Experiment
  8. Dataset
  9. Baselines
  10. Main results
  11. Number of iterations
  12. Persona of the Critic
  13. Debating style
  14. Related work
  15. LLM-based evaluation
  16. ...and 36 more sections

Figures (2)

  • Figure 1: Overall framework of DEBATE. Numbers around the arrows correspond to the steps in Appendix \ref{['steps_debate']}. The figure illustrates an example of deriving a consistency score in summarization task.
  • Figure 2: All experimental results shown in this figure are obtained using DEBATE with GPT-4 on the SummEval dataset, illustrating the effect of $n$ (the number of maximum iterations) (left), agent persona (middle), and debating strategies (right) on model performance.'n' refers to the nubmer of debate iterations among multi-agents, and 'Both' refers to adopting tie-breaker and Devil's Advocate simultaneously. See Appendix \ref{['system_messages']} for experiment details.