Do I Really Know? Learning Factual Self-Verification for Hallucination Reduction

TL;DR

This paper tackles the persistent problem of factual hallucinations in large language models by arguing that mitigation should be trained into the model rather than applied at inference time. It introduces VeriFY, a training-time framework that teaches models to reason about factual uncertainty through structured verification traces and a stage-level loss masking regime to avoid reinforcing hallucinated content. VeriFY constructs traces consisting of an initial answer, a probing verification question and answer, a revised answer, and a consistency judgment, using multiple verification strategies to probe factual validity. The authors also propose Hallucination F1 to evaluate the precision–recall trade-off under abstention, and demonstrate that VeriFY, especially with Span Masking and Minimal Masking, achieves substantial hallucination reduction with modest recall loss and strong cross-dataset generalization across model families and datasets.

Abstract

Factual hallucination remains a central challenge for large language models (LLMs). Existing mitigation approaches primarily rely on either external post-hoc verification or mapping uncertainty directly to abstention during fine-tuning, often resulting in overly conservative behavior. We propose VeriFY, a training-time framework that teaches LLMs to reason about factual uncertainty through consistency-based self-verification. VeriFY augments training with structured verification traces that guide the model to produce an initial answer, generate and answer a probing verification query, issue a consistency judgment, and then decide whether to answer or abstain. To address the risk of reinforcing hallucinated content when training on augmented traces, we introduce a stage-level loss masking approach that excludes hallucinated answer stages from the training objective while preserving supervision over verification behavior. Across multiple model families and scales, VeriFY reduces factual hallucination rates by 9.7 to 53.3 percent, with only modest reductions in recall (0.4 to 5.7 percent), and generalizes across datasets when trained on a single source. The source code, training data, and trained model checkpoints will be released upon acceptance.

Figures (3)

• Figure 1: Pareto frontiers of correct response rate vs. incorrect response rate for base models and their derived knowledge probing, self consistency, and VeriFY models for multiple model families. Knowledge probing and self consistency reduce hallucinations primarily by abstaining, which leads to a substantial drop in correct response rate. In contrast, VeriFY achieves large hallucination reductions with minor degradation to correct response rate.
• Figure 2: Verification trace and loss masking in VeriFY. (A) A structured verification trace in which a model produces an initial answer, generates a verification question to probe factual validity, answers the verification query, re-answers the original question, and emits a consistency judgment. (B) Verification strategies induce controlled semantic variations of the original query. (C) Training-time loss masking regimes, illustrating how hallucinated answer stages are selectively excluded from the objective while preserving supervision over verification and consistency-judgment.
• Figure 3: Effect of the number of distinct verification strategies used for the traces on F1 performance of VeriFY. For both Gemma-2-2B and Gemma-2-9B, performance improves as complementary strategies are added, with diminishing returns and mild degradation beyond 4, suggesting increased redundancy and noise.