User-Centric Evidence Ranking for Attribution and Fact Verification

TL;DR

This work formulates Evidence Ranking as a user-centric alternative to traditional evidence selection for attribution and fact verification, introducing Minimal Sufficient Rank (MSR) and IR-inspired metrics to minimize reading effort while preserving verifiability. It benchmarks one-shot and incremental approaches across embedding, NLI, reasoning-based rerankers, and large language models, demonstrating that incremental LLM-based ranking achieves the strongest overall performance, with substantial gains in both efficiency and verification accuracy. A unified benchmark integrating FEVER, HoVer, and WiCE reveals persistent headroom and practical challenges, while a controlled user study confirms that evidence ranking reduces reading load and improves decision quality compared to standard evidence selection. The findings highlight the potential of user-aligned, incremental ranking to produce interpretable and efficient verification systems, guiding future work toward model specialization, coherence considerations, and handling complex evidence landscapes.

Abstract

Attribution and fact verification are critical challenges in natural language processing for assessing information reliability. While automated systems and Large Language Models (LLMs) aim to retrieve and select concise evidence to support or refute claims, they often present users with either insufficient or overly redundant information, leading to inefficient and error-prone verification. To address this, we propose Evidence Ranking, a novel task that prioritizes presenting sufficient information as early as possible in a ranked list. This minimizes user reading effort while still making all available evidence accessible for sequential verification. We compare two approaches for the new ranking task: one-shot ranking and incremental ranking. We introduce a new evaluation framework, inspired by information retrieval metrics, and construct a unified benchmark by aggregating existing fact verification datasets. Extensive experiments with diverse models show that incremental ranking strategies better capture complementary evidence and that LLM-based methods outperform shallower baselines, while still facing challenges in balancing sufficiency and redundancy. Compared to evidence selection, we conduct a controlled user study and demonstrate that evidence ranking both reduces reading effort and improves verification. This work provides a foundational step toward more interpretable, efficient, and user-aligned information verification systems.

Figures (12)

• Figure 1: Illustration of the Evidence Ranking task. Given a claim and candidate evidence sentences, the system ranks the sentences so that users can stop reading once sufficient evidence is observed. Our evaluation simulates this process by measuring the number of sentences a user would read. Green sentences denote relevant evidence, and yellow denotes redundant evidence that can be skipped.
• Figure 2: Illustration of our proposed incremental ranking approach. (A) The naive one-shot algorithm produces a complete, global ranking of all evidence sentences in a single forward pass. (B) The incremental algorithm iteratively builds the ranking.
• Figure 3: Proportion of verified instances as a function of the number of sentences read (non-cumulative).
• Figure 4: Example prompt for LLM evidence ranking. The instructions are depicted in green, input to the model in black, and model's output in red.
• Figure 5: Example prompt for the first step in the incremental LLM evidence ranking, in which we don't have any selected sentences yet. The instructions are depicted in green, input to the model in black, and model's output in red. This prompt is intended for selecting the first sentence freely, without restrictions, i.e., it does not depend on any previously chosen sentences.