Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Attributed Text Generation of Large Language Models via Preference Learning

Dongfang Li, Zetian Sun, Baotian Hu, Zhenyu Liu, Xinshuo Hu, Xuebo Liu, Min Zhang

TL;DR

This work tackles the problem of unreliable content in large language models by reframing attribution as a preference-learning task. It introduces Automatic Preference Optimization (APO), combining a curated post-training dataset with a large automatically generated preference dataset and a progressive, experience-replay-enabled optimization strategy to reduce both generation and attribution hallucinations. Key contributions include being the first to apply preference learning to attribution tasks, constructing and releasing comprehensive data pipelines, and demonstrating state-of-the-art citation F1 and improved answer quality across ASQA, StrategyQA, and ELI5. The approach offers a practical, data-efficient pathway to produce more credible, citation-grounded LLM outputs, with broad implications for verification-centric NLP applications.

Abstract

Large language models have been widely adopted in natural language processing, yet they face the challenge of generating unreliable content. Recent works aim to reduce misinformation and hallucinations by resorting to attribution as a means to provide evidence (i.e., citations). However, current attribution methods usually focus on the retrieval stage and automatic evaluation that neglect mirroring the citation mechanisms in human scholarly writing to bolster credibility. In this paper, we address these challenges by modelling the attribution task as preference learning and introducing an Automatic Preference Optimization (APO) framework. First, we create a curated collection for post-training with 6,330 examples by collecting and filtering from existing datasets. Second, considering the high cost of labelling preference data, we further propose an automatic method to synthesize attribution preference data resulting in 95,263 pairs. Moreover, inspired by the human citation process, we further propose a progressive preference optimization method by leveraging fine-grained information. Extensive experiments on three datasets (i.e., ASQA, StrategyQA, and ELI5) demonstrate that APO achieves state-of-the-art citation F1 with higher answer quality.

Improving Attributed Text Generation of Large Language Models via Preference Learning

TL;DR

This work tackles the problem of unreliable content in large language models by reframing attribution as a preference-learning task. It introduces Automatic Preference Optimization (APO), combining a curated post-training dataset with a large automatically generated preference dataset and a progressive, experience-replay-enabled optimization strategy to reduce both generation and attribution hallucinations. Key contributions include being the first to apply preference learning to attribution tasks, constructing and releasing comprehensive data pipelines, and demonstrating state-of-the-art citation F1 and improved answer quality across ASQA, StrategyQA, and ELI5. The approach offers a practical, data-efficient pathway to produce more credible, citation-grounded LLM outputs, with broad implications for verification-centric NLP applications.

Abstract

Large language models have been widely adopted in natural language processing, yet they face the challenge of generating unreliable content. Recent works aim to reduce misinformation and hallucinations by resorting to attribution as a means to provide evidence (i.e., citations). However, current attribution methods usually focus on the retrieval stage and automatic evaluation that neglect mirroring the citation mechanisms in human scholarly writing to bolster credibility. In this paper, we address these challenges by modelling the attribution task as preference learning and introducing an Automatic Preference Optimization (APO) framework. First, we create a curated collection for post-training with 6,330 examples by collecting and filtering from existing datasets. Second, considering the high cost of labelling preference data, we further propose an automatic method to synthesize attribution preference data resulting in 95,263 pairs. Moreover, inspired by the human citation process, we further propose a progressive preference optimization method by leveraging fine-grained information. Extensive experiments on three datasets (i.e., ASQA, StrategyQA, and ELI5) demonstrate that APO achieves state-of-the-art citation F1 with higher answer quality.
Paper Structure (44 sections, 9 equations, 2 figures, 15 tables, 1 algorithm)

This paper contains 44 sections, 9 equations, 2 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text Generation for Verification
  4. Preference Optimization Methods
  5. Preliminary
  6. Pretraining and SFT Phase
  7. Preference Data Collection Phase
  8. Preference Optimization Phase
  9. Reinforcement Learning from Human Feedback (RLHF)
  10. Direct Preference Optimization (DPO)
  11. Methodology
  12. Problem Formulation
  13. Overall Framework
  14. Post-training
  15. Data Collection
  16. ...and 29 more sections

Figures (2)

  • Figure 1: A brief overview of our APO framework. We show APO in more detail in Figure \ref{['fig:framework']}.
  • Figure 2: The overall framework of the APO.