DebateQA: Evaluating Question Answering on Debatable Knowledge

TL;DR

DebateQA tackles the challenge of evaluating QA systems on inherently debatable knowledge by introducing a curated dataset of 2,941 questions paired with multiple POV-based partial answers. It defines two core metrics, Perspective Diversity (P.D.) and Dispute Awareness (D.A.), to assess how comprehensively models cover viewpoints and acknowledge debate, respectively. The authors validate that these metrics align with human judgments and are stable across backbones and prompts, applying them to 12 LLMs and RAG setups to reveal strengths in recognizing debate but variability in delivering diverse, well-grounded perspectives. Key findings include strong performance of open-source LLMs, mixed effects of RAG, and the importance of prompts and generation length in driving richer, more balanced debatable responses. The work provides a practical evaluation framework with broad implications for improving debatable knowledge interactions in real-world chat systems and highlights directions for future enhancements in domain grounding and perspective-aware generation.

Abstract

The rise of large language models (LLMs) has enabled us to seek answers to inherently debatable questions on LLM chatbots, necessitating a reliable way to evaluate their ability. However, traditional QA benchmarks assume fixed answers are inadequate for this purpose. To address this, we introduce DebateQA, a dataset of 2,941 debatable questions, each accompanied by multiple human-annotated partial answers that capture a variety of perspectives. We develop two metrics: Perspective Diversity, which evaluates the comprehensiveness of perspectives, and Dispute Awareness, which assesses if the LLM acknowledges the question's debatable nature. Experiments demonstrate that both metrics align with human preferences and are stable across different underlying models. Using DebateQA with two metrics, we assess 12 popular LLMs and retrieval-augmented generation methods. Our findings reveal that while LLMs generally excel at recognizing debatable issues, their ability to provide comprehensive answers encompassing diverse perspectives varies considerably.

Figures (15)

• Figure 1: Pipeline for curating DebateQA. The three main components of the pipeline are highlighted in different colors: sourcing debatable questions, collecting partial answers, and human annotation. Primary sources or tools used at each step are highlighted in bold.
• Figure 2: Correlation among annotators assessed by Spearman's $\rho$.
• Figure 3: Kendall's $\tau$ correlations of the P.D. metric using different backbone models.
• Figure 4: Kendall's $\tau$ correlations of the P.D. metric using different prompts.
• Figure 5: Agreements of the D.A. metric across different backbone models.