Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Post-training an LLM for RAG? Train on Self-Generated Demonstrations

Matthew Finlayson, Ilia Kulikov, Daniel M. Bikel, Barlas Oguz, Xilun Chen, Aasish Pappu

TL;DR

This work addresses knowledge-intensive QA with LLMs by examining retrieval-augmented generation (RAG) post-training. It introduces self-demo retrieval-augmented instruction tuning (SD-RA-IT), which generates and filters self-demonstrations conditioned on retrieved content to align training with the model’s distribution and to teach abstention on uncertain questions. SD-RA-IT improves precision, recall, and F1 while preserving non-RAG performance and reducing false refusals, outperforming gold-demonstration RA-IT across model sizes and retrieval counts. The findings suggest that leveraging self-generated supervision can mitigate out-of-distribution and hallucination issues in RAG systems and has practical implications for deploying robust RAG-enabled LLMs in knowledge-intensive tasks.

Abstract

Large language models (LLMs) often struggle with knowledge intensive NLP tasks, such as answering "Who won the latest World Cup?" because the knowledge they learn during training may be insufficient or outdated. Conditioning generation on retrieved documents -- a technique known as retrieval augmented generation (RAG) -- mitigates these shortcomings by allowing the model to leverage in-context information. Practitioners can improve LLM RAG performance by fine-tuning on retrieval-augmented instructions, but must beware that this can cause undesirable model behaviors like hallucinations. We attribute this degradation to the fact that the training data is likely to be out-of-distribution for the model and may suffer from quality issues, such as misalignment between retrievals and target responses (since retrievals are frequently added post-hoc). We propose a recipe for training RAG-enabled LLMs using self-generated demonstrations, thereby avoiding training on out-of-distribution text and integrating retrievals into the LLM responses. We evaluate our method on knowledge intensive question answering (QA) tasks and show that our method teaches LLMs to properly handle in-context retrievals and abstain from questions it will likely get wrong. Compared to conventional RA-IT methods, our method prevents model degradation in non-RAG settings while exhibiting superior QA performance.

Post-training an LLM for RAG? Train on Self-Generated Demonstrations

TL;DR

This work addresses knowledge-intensive QA with LLMs by examining retrieval-augmented generation (RAG) post-training. It introduces self-demo retrieval-augmented instruction tuning (SD-RA-IT), which generates and filters self-demonstrations conditioned on retrieved content to align training with the model’s distribution and to teach abstention on uncertain questions. SD-RA-IT improves precision, recall, and F1 while preserving non-RAG performance and reducing false refusals, outperforming gold-demonstration RA-IT across model sizes and retrieval counts. The findings suggest that leveraging self-generated supervision can mitigate out-of-distribution and hallucination issues in RAG systems and has practical implications for deploying robust RAG-enabled LLMs in knowledge-intensive tasks.

Abstract

Large language models (LLMs) often struggle with knowledge intensive NLP tasks, such as answering "Who won the latest World Cup?" because the knowledge they learn during training may be insufficient or outdated. Conditioning generation on retrieved documents -- a technique known as retrieval augmented generation (RAG) -- mitigates these shortcomings by allowing the model to leverage in-context information. Practitioners can improve LLM RAG performance by fine-tuning on retrieval-augmented instructions, but must beware that this can cause undesirable model behaviors like hallucinations. We attribute this degradation to the fact that the training data is likely to be out-of-distribution for the model and may suffer from quality issues, such as misalignment between retrievals and target responses (since retrievals are frequently added post-hoc). We propose a recipe for training RAG-enabled LLMs using self-generated demonstrations, thereby avoiding training on out-of-distribution text and integrating retrievals into the LLM responses. We evaluate our method on knowledge intensive question answering (QA) tasks and show that our method teaches LLMs to properly handle in-context retrievals and abstain from questions it will likely get wrong. Compared to conventional RA-IT methods, our method prevents model degradation in non-RAG settings while exhibiting superior QA performance.

Paper Structure

This paper contains 19 sections, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Training RAG-enabled LLMs
  4. Distilling prompted behavior
  5. Self-demo training & learning to abstain
  6. Method and Implementation
  7. Models, infrastructure, and datasets
  8. Automatic prompt generation
  9. Generating, filtering, and training on self-demonstrations
  10. Experimental details
  11. Benchmarks and Baselines
  12. Metrics
  13. Analysis and Results
  14. Precision
  15. Recall
  16. ...and 4 more sections

Figures (3)

  • Figure 1:
  • Figure 2:
  • Figure 3: A categorization of test instances. The box is divided into questions that the LLM can and cannot answer correctly, while the oval represents questions that the LLM attempts. Since we are more interested in reliability than raw accuracy on downstream tasks, we use precision ($\text{\# correct}/\text{\# attempted}$) and recall ($\text{\# correct}/\text{\# answerable}$) as our main metrics.