Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning

Shuzheng Si, Haozhe Zhao, Cheng Gao, Yuzhuo Bai, Zhitong Wang, Bofei Gao, Kangyang Luo, Wenhao Li, Yufei Huang, Gang Chen, Fanchao Qi, Minjia Zhang, Baobao Chang, Maosong Sun

TL;DR

The paper introduces CANOE, a post-training framework to enhance context-faithfulness in large language models without human annotations. It combines synthetic short-form QA data generated from Wikidata triples with Dual-GRPO, a rule-based reinforcement learning approach that assigns separate rewards to long-form and short-form outputs, including accuracy, proxy, and format rewards. Evaluations across 11 downstream tasks show significant faithfulness gains, with Canoe outperforming several state-of-the-art models in average metrics and demonstrating robustness in retrieval-augmented generation and multilingual transfer. Ablation studies confirm the necessity of the dual-reward design and diverse synthetic tasks, while analysis highlights CANOE’s potential to reduce faithfulness hallucinations without reliance on human-annotated data.

Abstract

Teaching large language models (LLMs) to be faithful in the provided context is crucial for building reliable information-seeking systems. Therefore, we propose a systematic framework, CANOE, to reduce faithfulness hallucinations of LLMs across different downstream tasks without human annotations. Specifically, we first synthesize short-form question-answering (QA) data with four diverse tasks to construct high-quality and easily verifiable training data without human annotation. Also, we propose Dual-GRPO, a rule-based reinforcement learning method that includes three tailored rule-based rewards derived from synthesized short-form QA data, while simultaneously optimizing both short-form and long-form response generation. Notably, Dual-GRPO eliminates the need to manually label preference data to train reward models and avoids over-optimizing short-form generation when relying only on the synthesized short-form QA data. Experimental results show that CANOE greatly improves the faithfulness of LLMs across 11 different tasks, even outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1.

Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning

TL;DR

The paper introduces CANOE, a post-training framework to enhance context-faithfulness in large language models without human annotations. It combines synthetic short-form QA data generated from Wikidata triples with Dual-GRPO, a rule-based reinforcement learning approach that assigns separate rewards to long-form and short-form outputs, including accuracy, proxy, and format rewards. Evaluations across 11 downstream tasks show significant faithfulness gains, with Canoe outperforming several state-of-the-art models in average metrics and demonstrating robustness in retrieval-augmented generation and multilingual transfer. Ablation studies confirm the necessity of the dual-reward design and diverse synthetic tasks, while analysis highlights CANOE’s potential to reduce faithfulness hallucinations without reliance on human-annotated data.

Abstract

Teaching large language models (LLMs) to be faithful in the provided context is crucial for building reliable information-seeking systems. Therefore, we propose a systematic framework, CANOE, to reduce faithfulness hallucinations of LLMs across different downstream tasks without human annotations. Specifically, we first synthesize short-form question-answering (QA) data with four diverse tasks to construct high-quality and easily verifiable training data without human annotation. Also, we propose Dual-GRPO, a rule-based reinforcement learning method that includes three tailored rule-based rewards derived from synthesized short-form QA data, while simultaneously optimizing both short-form and long-form response generation. Notably, Dual-GRPO eliminates the need to manually label preference data to train reward models and avoids over-optimizing short-form generation when relying only on the synthesized short-form QA data. Experimental results show that CANOE greatly improves the faithfulness of LLMs across 11 different tasks, even outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1.

Paper Structure

This paper contains 26 sections, 6 equations, 25 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Training Data Construction
  4. Reinforcement Learning Protocol
  5. Reward Design
  6. Experiments
  7. Tasks and Datasets
  8. Baselines and Implementation Details
  9. Main Results
  10. Analysis
  11. Related Work
  12. Conclusion
  13. Training Data Details
  14. Triples from Wikidata
  15. Construction of Four Different Tasks
  16. ...and 11 more sections

Figures (25)

  • Figure 1: Average score on 11 downstream tasks vs model size. With only 7B parameters, Canoe already exceeds state-of-the-art LLMs like GPT-4o and OpenAI o1.
  • Figure 2: An overview of Canoe framework. Canoe first synthesizes easily verifiable short-form QA data and then proposes the Dual-GRPO with designed rule-based rewards to improve the faithfulness of LLMs.
  • Figure 3: Model performance comparison on FaithEval in a closed-book QA setting and counterfactual context setting. Our models are colored in orange. We report the results from the chat version of LLaMA-3 and Qwen-2.5.
  • Figure 4: The average perplexity score of 110 negative samples for each model from eleven datasets.
  • Figure 5: Human evaluation across four key dimensions.
  • ...and 20 more figures