DiVA: Fine-grained Factuality Verification with Agentic-Discriminative Verifier

TL;DR

DiVA addresses the challenge of fine-grained factuality verification for LLMs by uniting agentic search with a discriminative scorer, enabling continuous scoring and external knowledge grounding. It introduces FGVeriBench, a ranking-based benchmark spanning general and multi-hop tasks to evaluate nuanced factuality. Empirical results show DiVA outperforms baselines across binary, long-form, and generation settings, with ablations highlighting the importance of context compression and diverse knowledge sources. The work offers a practical pathway to more reliable LLM outputs and establishes FGVeriBench as a robust testbed for future factuality research.

Abstract

Despite the significant advancements of Large Language Models (LLMs), their factuality remains a critical challenge, fueling growing interest in factuality verification. Existing research on factuality verification primarily conducts binary judgments (e.g., correct or incorrect), which fails to distinguish varying degrees of error severity. This limits its utility for applications such as fine-grained evaluation and preference optimization. To bridge this gap, we propose the Agentic Discriminative Verifier (DiVA), a hybrid framework that synergizes the agentic search capabilities of generative models with the precise scoring aptitude of discriminative models. We also construct a new benchmark, FGVeriBench, as a robust testbed for fine-grained factuality verification. Experimental results on FGVeriBench demonstrate that our DiVA significantly outperforms existing methods on factuality verification for both general and multi-hop questions.

Figures (13)

• Figure 1: Illustrative comparison of binary and fine-grained factuality verification. Notice binary verification fails to distinguish varying degrees of error severity.
• Figure 2: The whole pipeline of factuality verification with DiVA. Left is the inference process, which consists of three steps, and right is the data construction and training process of the discriminative module of DiVA.
• Figure 3: The comparison of different verifier architectures. We compare the different architectures in for aspects: agentic reasoning ability, accessibility to external knowledge, fine-grained scoring ability and training efficiency.
• Figure 4: Construction process of FGVeriBench.
• Figure 5: Model distribution of different answer ranks in FGVeriBench. Rank 1 is with best factuality.