MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs

TL;DR

This work tackles the challenge of factual accuracy in long-form LLM outputs by introducing LongHalluQA, a Chinese long-form factuality benchmark, and MAD-Fact, a multi-agent debate framework that decomposes, verifies, and adjudicates atomic factual claims. It couples a fact-importance hierarchy with weighted evaluation metrics to reflect the differential significance of claims, and validates the approach with extensive experiments on LongFact and LongHalluQA, showing that larger and Chinese-focused systems perform better on their native tasks while cross-domain accuracy remains challenging. The MAD-Fact architecture—comprising Clerk, Jury, and Judge—mitigates single-model biases via retrieval-enabled debates and role-based analyses, achieving superior factuality scores over strong baselines such as SAFE and FIRE. The results underscore the potential of hierarchical, multi-agent, and retrieval-informed evaluation to guide safer deployment of LLMs in high-stakes domains, while also outlining practical paths to improve efficiency, grounding, and robustness.

Abstract

The widespread adoption of Large Language Models (LLMs) raises critical concerns about the factual accuracy of their outputs, especially in high-risk domains such as biomedicine, law, and education. Existing evaluation methods for short texts often fail on long-form content due to complex reasoning chains, intertwined perspectives, and cumulative information. To address this, we propose a systematic approach integrating large-scale long-form datasets, multi-agent verification mechanisms, and weighted evaluation metrics. We construct LongHalluQA, a Chinese long-form factuality dataset; and develop MAD-Fact, a debate-based multi-agent verification system. We introduce a fact importance hierarchy to capture the varying significance of claims in long-form texts. Experiments on two benchmarks show that larger LLMs generally maintain higher factual consistency, while domestic models excel on Chinese content. Our work provides a structured framework for evaluating and enhancing factual reliability in long-form LLM outputs, guiding their safe deployment in sensitive domains.

Figures (13)

• Figure 1: Examples of core factual claims and auxiliary claims. Both responses from LLM Alice and LLM Bob contain a single incorrect atomic claim. However, from the user’s perspective, the answer provided by LLM Bob is clearly inferior, as its error occurs within a core factual claim, whereas LLM Alice's mistake affects only an auxiliary claim.
• Figure 2: The construction process of the LongHalluQA dataset.
• Figure 3: The topic distribution of the LongHalluQA dataset.
• Figure 4: Comparison of samples between HalluQA and LongHalluQA.
• Figure 5: The overall framework of the MAD-Fact system.