FLAME: Factuality-Aware Alignment for Large Language Models

TL;DR

The paper tackles hallucination in large language model alignment by proposing Flame, a factuality-aware alignment framework that integrates fact-based instruction classification with separate factuality and instruction-following rewards. It extends baseline SFT and Direct Preference Optimization (DPO) training with two reward models: RM_IF for instruction-following and RM_fact for factuality verification, using self-generated data to avoid introducing unknown knowledge. Across Biography, Alpaca Eval, and TruthfulQA benchmarks, Flame achieves higher factuality scores (FactScore) while preserving or improving instruction-following performance, outperforming standard SFT+ DPO baselines. Ablation studies reveal the importance of categorizing fact-based instructions and relying on self-generated, fact-grounded data; limitations include focusing on two alignment skills and potential trade-offs between factuality and helpfulness. The approach offers a practical path toward reducing hallucinations in real-world instruction-tuning scenarios and informs future multi-skill alignment research.

Abstract

Alignment is a standard procedure to fine-tune pre-trained large language models (LLMs) to follow natural language instructions and serve as helpful AI assistants. We have observed, however, that the conventional alignment process fails to enhance the factual accuracy of LLMs, and often leads to the generation of more false facts (i.e. hallucination). In this paper, we study how to make the LLM alignment process more factual, by first identifying factors that lead to hallucination in both alignment steps:\ supervised fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that training the LLM on new knowledge or unfamiliar texts can encourage hallucination. This makes SFT less factual as it trains on human labeled data that may be novel to the LLM. Furthermore, reward functions used in standard RL can also encourage hallucination, because it guides the LLM to provide more helpful responses on a diverse set of instructions, often preferring longer and more detailed responses. Based on these observations, we propose factuality-aware alignment, comprised of factuality-aware SFT and factuality-aware RL through direct preference optimization. Experiments show that our proposed factuality-aware alignment guides LLMs to output more factual responses while maintaining instruction-following capability.

Figures (11)

• Figure 1: Models' helpfulness on Alpaca Eval vs factuality on biography. Helpfulness is measured by models' win rate over our baseline $\mathrm{SFT}$ + $\mathrm{DPO}$ on Alpaca Eval. Dot size represents averaged length of bio generation.
• Figure 2: Instructions from Open Assistant dataset. The instructions are classified with $\mathrm{SFT}$ model using the prompt in Appendix, Figure \ref{['fig:prompt_for_instruction_classifier']}.
• Figure 3: Illustration of response generation using a pre-trained LLM ($\mathrm{PT}$) with few-shot demonstration.
• Figure 4: Illustration of factuality-aware alignment.
• Figure 5: Prompt to check whether an instruction is fact-based.