Reducing Hallucinations in LLMs via Factuality-Aware Preference Learning

TL;DR

This paper tackles the persistent challenge of hallucinations in LLMs by introducing F-DPO, a factuality-aware extension of Direct Preference Optimization. By using binary factuality labels and two mechanisms—label flipping and a factuality-margin—the method corrects misordered preference pairs and amplifies signals that distinguish factual from hallucinated responses, all in a single-stage training setup without auxiliary reward models. Empirical results across seven open-weight models (1B–14B) show substantial reductions in hallucination rates and improved factuality scores, with strong generalization to TruthfulQA benchmarks. The findings highlight the importance of explicit factuality supervision in preference learning and demonstrate a scalable, resource-efficient path to safer, more truthful LLM outputs.

Abstract

Preference alignment methods such as RLHF and Direct Preference Optimization (DPO) improve instruction following, but they can also reinforce hallucinations when preference judgments reward fluency and confidence over factual correctness. We introduce F-DPO (Factuality-aware Direct Preference Optimization), a simple extension of DPO that uses only binary factuality labels. F-DPO (i) applies a label-flipping transformation that corrects misordered preference pairs so the chosen response is never less factual than the rejected one, and (ii) adds a factuality-aware margin that emphasizes pairs with clear correctness differences, while reducing to standard DPO when both responses share the same factuality. We construct factuality-aware preference data by augmenting DPO pairs with binary factuality indicators and synthetic hallucinated variants. Across seven open-weight LLMs (1B-14B), F-DPO consistently improves factuality and reduces hallucination rates relative to both base models and standard DPO. On Qwen3-8B, F-DPO reduces hallucination rates by five times (from 0.424 to 0.084) while improving factuality scores by 50 percent (from 5.26 to 7.90). F-DPO also generalizes to out-of-distribution benchmarks: on TruthfulQA, Qwen2.5-14B achieves plus 17 percent MC1 accuracy (0.500 to 0.585) and plus 49 percent MC2 accuracy (0.357 to 0.531). F-DPO requires no auxiliary reward model, token-level annotations, or multi-stage training.

Figures (4)

• Figure 1: Overview of F-DPO.Left: The data pipeline constructs factuality-aware preference pairs by combining cleaned human data with synthetic generations and automated factuality evaluation, followed by transformation, merging, and balancing. Center: Factuality alignment is achieved through label flipping, which enforces factual ordering between preferred and dispreferred responses and defines a factuality margin based on label differences. Right: Preference optimization applies a modified DPO objective that augments standard preference learning with a factuality-aware margin penalty to explicitly discourage hallucinated responses.
• Figure 2: F-DPO data construction pipeline. Binary factuality labels from GPT-4o-mini are assigned to Skywork preference pairs. Synthetic hallucinated variants are generated, merged, and balanced across configurations $(h_w, h_l)$. Label-flipping ensures chosen responses are never less factual than rejected ones.
• Figure 3: Baseline (Default DPO) vs. $\lambda$-tuned rewards across models.
• Figure 4: Qwen2.5-14B: Effect of factuality penalty strength $\lambda$ on model performance.