Beyond Under-Alignment: Atomic Preference Enhanced Factuality Tuning for Large Language Models

TL;DR

This paper conducts a comprehensive evaluation of the factuality of different models tuned by various preference learning algorithms and demonstrates that their performance on OOD datasets either increases minimally or decreases, and proposes APEFT, a framework that enhances model's awareness of factuality at the granularity of individual facts.

Abstract

Large language models (LLMs) have achieved remarkable success but still tend to generate factually erroneous responses, a phenomenon known as hallucination. A recent trend is to use preference learning to fine-tune models to align with factuality. However, existing work primarily evaluates fine-tuned models on in-domain (ID) datasets and the factuality on out-of-domain (OOD) datasets remains underexplored. In this paper, we conduct a comprehensive evaluation of the factuality of different models tuned by various preference learning algorithms and demonstrate that their performance on OOD datasets either increases minimally or decreases. Subsequently, we reveal that the main cause of model's failure to uphold factuality under a distribution shift is \textbf{under-alignment}, rather than \textbf{over-alignment}, by analyzing the token distribution shift of the models before and after tuning. Finally, we propose \textbf{APEFT} (\textbf{A}tomic \textbf{P}reference \textbf{E}nhanced \textbf{F}actuality \textbf{T}uning), a framework that enhances model's awareness of factuality at the granularity of individual facts. Extensive experiments demonstrate that APEFT improves model performance by an average of $\boldsymbol{3.45\%}$ on both ID and OOD datasets, which is highly effective.

Figures (15)

• Figure 1: Illustration of our work. The combination of general preferences and atomic preferences can enhance model factuality on OOD queries across various preference learning algorithms.
• Figure 2: Performance changes of LLaMA-3-8B-Instruct before and after tuning using various preference learning techniques.
• Figure 3: Token shift analysis on LLaMA-2 trained by DPO. More results are presented in Appendix \ref{['sec:token_shift_analysis']}.
• Figure 4: Illustration of our proposed APEFT. It creates atomic preferences from the responses in original general preferences to enhance model's awareness of factuality at the granularity of individual facts.
• Figure 5: Experimental results of models trained by various number of training preferences.