Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Atomic Consistency Preference Optimization for Long-Form Question Answering

Jingfeng Chen, Raghuveer Thirukovalluru, Junlin Wang, Kaiwei Luo, Bhuwan Dhingra

TL;DR

This work tackles the challenge of factual hallucinations in long-form QA by proposing Atomic Consistency Preference Optimization (ACPO), a fully self-supervised alignment method that relies on atomic self-consistency signals rather than external knowledge bases or stronger models. ACPO generates multiple stochastic outputs, decomposes them into atomic facts, clusters these facts to identify consistent versus non-consistent information, and creates preference data used to fine-tune via Direct Preference Optimization (DPO). Through extensive experiments on LongFact and BioGen, ACPO surpasses a strong supervised baseline (FactAlign) by an average of 1.95 points in factual precision, while offering orders-of-magnitude improvements in efficiency for constructing preference data. The approach demonstrates that self-supervised alignment can meaningfully improve factual reliability in long-form QA without external supervision, enabling privacy-preserving and scalable deployment.

Abstract

Large Language Models (LLMs) often produce factoid hallucinations - plausible yet incorrect answers. A common mitigation strategy is model alignment, which improves factual accuracy by training on curated (factual, non-factual) pairs. However, this approach often relies on a stronger model (e.g., GPT-4) or an external knowledge base to assess factual correctness that may not always be accessible. Addressing this, we propose Atomic Consistency Preference Optimization (ACPO), a self-supervised preference-tuning method that enhances factual accuracy without external supervision. ACPO leverages atomic consistency signals (i.e., the agreement of individual facts across multiple stochastic responses) to identify high- and low-quality data pairs for model alignment. Despite being fully self-supervised, ACPO outperforms the strong supervised alignment baseline by 1.95 points averaged across Phi-3 and Llama3 on the LongFact and BioGen datasets, demonstrating its effectiveness in improving factual reliability without relying on external models or knowledge bases.

Atomic Consistency Preference Optimization for Long-Form Question Answering

TL;DR

This work tackles the challenge of factual hallucinations in long-form QA by proposing Atomic Consistency Preference Optimization (ACPO), a fully self-supervised alignment method that relies on atomic self-consistency signals rather than external knowledge bases or stronger models. ACPO generates multiple stochastic outputs, decomposes them into atomic facts, clusters these facts to identify consistent versus non-consistent information, and creates preference data used to fine-tune via Direct Preference Optimization (DPO). Through extensive experiments on LongFact and BioGen, ACPO surpasses a strong supervised baseline (FactAlign) by an average of 1.95 points in factual precision, while offering orders-of-magnitude improvements in efficiency for constructing preference data. The approach demonstrates that self-supervised alignment can meaningfully improve factual reliability in long-form QA without external supervision, enabling privacy-preserving and scalable deployment.

Abstract

Large Language Models (LLMs) often produce factoid hallucinations - plausible yet incorrect answers. A common mitigation strategy is model alignment, which improves factual accuracy by training on curated (factual, non-factual) pairs. However, this approach often relies on a stronger model (e.g., GPT-4) or an external knowledge base to assess factual correctness that may not always be accessible. Addressing this, we propose Atomic Consistency Preference Optimization (ACPO), a self-supervised preference-tuning method that enhances factual accuracy without external supervision. ACPO leverages atomic consistency signals (i.e., the agreement of individual facts across multiple stochastic responses) to identify high- and low-quality data pairs for model alignment. Despite being fully self-supervised, ACPO outperforms the strong supervised alignment baseline by 1.95 points averaged across Phi-3 and Llama3 on the LongFact and BioGen datasets, demonstrating its effectiveness in improving factual reliability without relying on external models or knowledge bases.
Paper Structure (40 sections, 2 equations, 1 figure, 12 tables)

This paper contains 40 sections, 2 equations, 1 figure, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Inference Time Methods
  4. Using Retrievers, Self-Evaluation
  5. Using Self-Consistency
  6. Alignment Methods
  7. Background: DPO Alignment Mechanism
  8. Methodology
  9. Overview
  10. Data Generation
  11. Step 1: Initial Responses Generation
  12. Step 2: Splitting Initial Responses for Atomic Facts
  13. Step 3: Clustering Atomic Facts
  14. Step 4: Consistent (C) and Non-consistent (NC) Clusters
  15. Step 5: Scoring Function
  16. ...and 25 more sections

Figures (1)

  • Figure 1: ACPO data curation pipeline. Steps 1–5 (Top): Generate stochastic responses for a question, Split and extract atomic facts; Cluster the atomic facts; Identify consistent and non-consistent clusters; Score responses based on cluster consistency. Step 6-7 (Bottom): Highest and lowest scoring curated as preference pairs; DPO alignment.