Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teaching Language Models to Hallucinate Less with Synthetic Tasks

Erik Jones, Hamid Palangi, Clarisse Simões, Varun Chandrasekaran, Subhabrata Mukherjee, Arindam Mitra, Ahmed Awadallah, Ece Kamar

TL;DR

SynTra tackles the challenge of hallucinations in abstractive summarization by introducing a controllable synthetic task with an easy-to-measure hallucination signal. It optimizes the LLM system message through prefix-tuning on this synthetic task, with optional regularization from reference data, and transfers the learned instructions to real tasks such as MS MARCO, QMSum, and ACI-Bench. Across two 13B models, SynTra reduces hallucination and improves grounding without sacrificing output quality, and in some cases system-message optimization outperforms full-model fine-tuning. The results support the idea that synthetic data can be a scalable, targeted tool to mitigate undesired LLM behaviors, while also revealing the importance of regularization and transfer design. The work suggests promising directions for extending synthetic-task methods to broader model families and downstream tasks.

Abstract

Large language models (LLMs) frequently hallucinate on abstractive summarization tasks such as document-based question-answering, meeting summarization, and clinical report generation, even though all necessary information is included in context. However, optimizing LLMs to hallucinate less on these tasks is challenging, as hallucination is hard to efficiently evaluate at each optimization step. In this work, we show that reducing hallucination on a synthetic task can also reduce hallucination on real-world downstream tasks. Our method, SynTra, first designs a synthetic task where hallucinations are easy to elicit and measure. It next optimizes the LLM's system message via prefix-tuning on the synthetic task, and finally transfers the system message to realistic, hard-to-optimize tasks. Across three realistic abstractive summarization tasks, SynTra reduces hallucination for two 13B-parameter LLMs using only a synthetic retrieval task for supervision. We also find that optimizing the system message rather than the model weights can be critical; fine-tuning the entire model on the synthetic task can counterintuitively increase hallucination. Overall, SynTra demonstrates that the extra flexibility of working with synthetic data can help mitigate undesired behaviors in practice.

Teaching Language Models to Hallucinate Less with Synthetic Tasks

TL;DR

SynTra tackles the challenge of hallucinations in abstractive summarization by introducing a controllable synthetic task with an easy-to-measure hallucination signal. It optimizes the LLM system message through prefix-tuning on this synthetic task, with optional regularization from reference data, and transfers the learned instructions to real tasks such as MS MARCO, QMSum, and ACI-Bench. Across two 13B models, SynTra reduces hallucination and improves grounding without sacrificing output quality, and in some cases system-message optimization outperforms full-model fine-tuning. The results support the idea that synthetic data can be a scalable, targeted tool to mitigate undesired LLM behaviors, while also revealing the importance of regularization and transfer design. The work suggests promising directions for extending synthetic-task methods to broader model families and downstream tasks.

Abstract

Large language models (LLMs) frequently hallucinate on abstractive summarization tasks such as document-based question-answering, meeting summarization, and clinical report generation, even though all necessary information is included in context. However, optimizing LLMs to hallucinate less on these tasks is challenging, as hallucination is hard to efficiently evaluate at each optimization step. In this work, we show that reducing hallucination on a synthetic task can also reduce hallucination on real-world downstream tasks. Our method, SynTra, first designs a synthetic task where hallucinations are easy to elicit and measure. It next optimizes the LLM's system message via prefix-tuning on the synthetic task, and finally transfers the system message to realistic, hard-to-optimize tasks. Across three realistic abstractive summarization tasks, SynTra reduces hallucination for two 13B-parameter LLMs using only a synthetic retrieval task for supervision. We also find that optimizing the system message rather than the model weights can be critical; fine-tuning the entire model on the synthetic task can counterintuitively increase hallucination. Overall, SynTra demonstrates that the extra flexibility of working with synthetic data can help mitigate undesired behaviors in practice.
Paper Structure (50 sections, 3 equations, 2 figures, 4 tables)

This paper contains 50 sections, 3 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. The SynTra pipeline
  3. Setup
  4. Synthetic tasks
  5. Reducing hallucination with SynTra
  6. Optimization.
  7. Regularizing with reference data.
  8. Full loss function.
  9. Evaluating SynTra
  10. Setup
  11. LLMs.
  12. Optimization details.
  13. Realistic tasks
  14. Search and retrieve (MS MARCO)
  15. Meeting summarization (QMSum)
  16. ...and 35 more sections

Figures (2)

  • Figure 1: Overview of the SynTra framework. We first define a synthetic task where hallucination is easy to tractably evaluate. Next, we optimize the LLM system message on this task by learning a continuous postfix via prefix-tuning. We then transfer the learned system message across real tasks.
  • Figure 2: Hallucination rate on the names retrieval task on the original LLM (Original) when optimizing the system message (Sys.) or full LLM weights (Model) on either just the synthetic data (synth.) or mixture of synthetic and reference data (synth. + ref.). We measure whether the LLM is exactly correct, and whether it only generates names on the original list (No new names).