FactCorrector: A Graph-Inspired Approach to Long-Form Factuality Correction of Large Language Models

TL;DR

The paper tackles persistent factuality errors in large language models by introducing FactCorrector, a post-hoc correction pipeline that uses a FactReasoner-based critic to decompose outputs into atomic facts, retrieve supporting or contradicting evidence, and reason over a graphical model to generate structured feedback for a refinement model. It also introduces VELI5, a large benchmark with injected factual errors to enable rigorous evaluation of long-form factuality correction, and demonstrates that FactCorrector yields reliable factuality gains across diverse datasets and base models, including open-source LLMs. The authors further explore short-horizon SFT corrections and provide a comprehensive human evaluation, highlighting generalization and robustness while acknowledging computational costs and dependencies on prompts and retrieval quality. Overall, the work provides a scalable, cross-domain approach to factuality correction with a well-founded evaluation framework and practical implications for safer LLM deployment.

Abstract

Large language models (LLMs) are widely used in knowledge-intensive applications but often generate factually incorrect responses. A promising approach to rectify these flaws is correcting LLMs using feedback. Therefore, in this paper, we introduce FactCorrector, a new post-hoc correction method that adapts across domains without retraining and leverages structured feedback about the factuality of the original response to generate a correction. To support rigorous evaluations of factuality correction methods, we also develop the VELI5 benchmark, a novel dataset containing systematically injected factual errors and ground-truth corrections. Experiments on VELI5 and several popular long-form factuality datasets show that the FactCorrector approach significantly improves factual precision while preserving relevance, outperforming strong baselines. We release our code at https://ibm.biz/factcorrector.

Figures (16)

• Figure 1: FactReasoner pipeline and an example graphical model with 5 atoms and 8 context variables.
• Figure 2: Illustration of the FactCorrector pipeline for long-form factuality correction of LLMs.
• Figure 3: Mean relative gains for factuality metrics across models on the Veli5 dataset.
• Figure 4: A graphical model with three bi-valued variables $X_1$, $X_2$ and $X_3$, and three binary functions.
• Figure 5: FactReasoner pipeline and an example graphical model with 3 atom and 6 context variables.

Theorems & Definitions (4)

• Example 1
• Example 2
• Example 3
• Example 4