Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models

Singon Kim

TL;DR

Retrieval-augmented language models struggle when retrieved documents contain noise, causing important information to be missed or misinformation to slip through. The authors propose ACoRN, a two-step training framework that (1) uses offline data augmentation to simulate retrieval noise and (2) finetunes the compressor to produce summaries centered on evidential content that directly supports the correct answer. By distinguishing evidential documents from noise and distilling labels from an evidential-focused teacher, ACoRN yields higher EM/F1 and preserves answer strings while reducing inference time on ODQA benchmarks. The approach demonstrates improved robustness to both irrelevant and factual-error documents, offering practical benefits for real-world retrieval-augmented systems. Overall, ACoRN provides a scalable, noise-robust abstractive compression method that enhances reliability and efficiency in RAG setups.

Abstract

Abstractive compression utilizes smaller langauge models to condense query-relevant context, reducing computational costs in retrieval-augmented generation (RAG). However, retrieved documents often include information that is either irrelevant to answering the query or misleading due to factual incorrect content, despite having high relevance scores. This behavior indicates that abstractive compressors are more likely to omit important information essential for the correct answer, especially in long contexts where attention dispersion occurs. To address this issue, we categorize retrieved documents in a more fine-grained manner and propose Abstractive Compression Robust against Noise (ACoRN), which introduces two novel training steps. First, we use offline data augmentation on the training dataset to enhance compressor robustness against two distinct types of retrieval noise. Second, since the language model based compressor cannot fully utilize information from multiple retrieved documents and exhibits positional bias, we perform finetuning to generate summaries centered around key information that directly supports the correct answer. Our experiments demonstrate that T5-large, trained with ACoRN as a compressor, improves EM and F1 scores while preserving the answer string, which could serve as direct evidence. ACoRN excels on datasets with many accuracy reducing documents, making it highly useful in real-world scenarios.

Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models

TL;DR

Retrieval-augmented language models struggle when retrieved documents contain noise, causing important information to be missed or misinformation to slip through. The authors propose ACoRN, a two-step training framework that (1) uses offline data augmentation to simulate retrieval noise and (2) finetunes the compressor to produce summaries centered on evidential content that directly supports the correct answer. By distinguishing evidential documents from noise and distilling labels from an evidential-focused teacher, ACoRN yields higher EM/F1 and preserves answer strings while reducing inference time on ODQA benchmarks. The approach demonstrates improved robustness to both irrelevant and factual-error documents, offering practical benefits for real-world retrieval-augmented systems. Overall, ACoRN provides a scalable, noise-robust abstractive compression method that enhances reliability and efficiency in RAG setups.

Abstract

Abstractive compression utilizes smaller langauge models to condense query-relevant context, reducing computational costs in retrieval-augmented generation (RAG). However, retrieved documents often include information that is either irrelevant to answering the query or misleading due to factual incorrect content, despite having high relevance scores. This behavior indicates that abstractive compressors are more likely to omit important information essential for the correct answer, especially in long contexts where attention dispersion occurs. To address this issue, we categorize retrieved documents in a more fine-grained manner and propose Abstractive Compression Robust against Noise (ACoRN), which introduces two novel training steps. First, we use offline data augmentation on the training dataset to enhance compressor robustness against two distinct types of retrieval noise. Second, since the language model based compressor cannot fully utilize information from multiple retrieved documents and exhibits positional bias, we perform finetuning to generate summaries centered around key information that directly supports the correct answer. Our experiments demonstrate that T5-large, trained with ACoRN as a compressor, improves EM and F1 scores while preserving the answer string, which could serve as direct evidence. ACoRN excels on datasets with many accuracy reducing documents, making it highly useful in real-world scenarios.

Paper Structure

This paper contains 22 sections, 9 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Abstractive Compression
  4. Noise-Robust Retrieval-Augmented Language Models
  5. Methodology
  6. Overview
  7. Classifying Noise Documents
  8. Noise Documents Construction
  9. Evidential Pseudo Labels for Summarization
  10. Abstractive Compression Training
  11. Experiments
  12. Implementation Details
  13. Baselines
  14. Datasets Construction
  15. Training
  16. ...and 7 more sections

Figures (3)

  • Figure 1: An illustrative example of a challenge in retrieving and summarizing information supporting to the correct answer from the documents. The compressor performs well in summarizing content supported to the correct answer when only the document including the correct answer is provided. However, it generates incorrect information or misses the key information when the retrieved documents contain inaccurate or irrelevant information.
  • Figure 2: Exact Match (EM) scores for different types of noise documents, including irrelevant documents and factual error documents. Flan-T5-large b29 compresses documents using Query-Focused Summarization (QFS), compressed passages are then passed to LLaMA-3.1-8B-Instruct b36 to generate answers to the queries.
  • Figure 3: Performance comparison of GPT-3.5-turbo QFS using only evidential documents versus all top-5 documents in the prompt, based on 100 random samples for each evidential document count $N$ in top-5 retrieval. The $N$=0 case (evidential only) means using only internal knowledge. The compressed output is passed to the inference model's $M$ prompt, LLaMA-3.1-8B-Instruct. The dotted line represents performance from 100 randomly sampled instances, regardless of $N$.