From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification
Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng
TL;DR
This work tackles fact verification by reframing evidence retrieval from relevance to utility, introducing the feedback-based evidence retriever (FER). FER employs a coarse PRP-based retrieval followed by a fine-grained retriever trained with verifier feedback, optimizing a two-term objective $L = a L_{cla} + b L_{uti}$ that combines ground-truth evidence plausibility and utility divergence measured by a verifier distribution $D_\phi$. On FEVER, FER achieves a substantial gain of 23.7% in F1 over state-of-the-art baselines, and improves verification results across multiple models using evidence retrieved by FER. The approach demonstrates that incorporating verifier feedback can selectively retrieve evidence that is both relevant and highly useful for verification, with implications for retrieval-enhanced FV systems and broader task-based IR.
Abstract
Retrieval-enhanced methods have become a primary approach in fact verification (FV); it requires reasoning over multiple retrieved pieces of evidence to verify the integrity of a claim. To retrieve evidence, existing work often employs off-the-shelf retrieval models whose design is based on the probability ranking principle. We argue that, rather than relevance, for FV we need to focus on the utility that a claim verifier derives from the retrieved evidence. We introduce the feedback-based evidence retriever(FER) that optimizes the evidence retrieval process by incorporating feedback from the claim verifier. As a feedback signal we use the divergence in utility between how effectively the verifier utilizes the retrieved evidence and the ground-truth evidence to produce the final claim label. Empirical studies demonstrate the superiority of FER over prevailing baselines.