Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tackling the Inherent Difficulty of Noise Filtering in RAG

Jingyu Liu, Jiaen Lin, Yong Liu

TL;DR

This work addresses the problem that noisy retrieved documents in retrieval-augmented generation can degrade LLM performance and induce hallucinations. It argues that filtering irrelevant content is inherently difficult for limited-capacity retrievers and that linear fine-tuning distorts attention patterns; to overcome this, the authors introduce a nonlinear attention rectification mechanism that adds a custom g(·) function to the attention update, enabling aggressive noise suppression while preserving useful reasoning. The approach is analyzed theoretically (the triple-wise nature of relevance) and validated empirically across five benchmarks, using LoRA fine-tuning and a query-before-document arrangement; results show robust improvements in noise robustness and overall accuracy. The work offers a practical path to strengthen RAG systems in real-world scenarios, by decoupling noise filtering from semantic reasoning through a carefully designed nonlinear attention update, and it highlights the potential benefits and limitations of tuning pre-trained LLMs rather than training from scratch.

Abstract

Retrieval-Augmented Generation (RAG) has become a widely adopted approach to enhance Large Language Models (LLMs) by incorporating external knowledge and reducing hallucinations. However, noisy or irrelevant documents are often introduced during RAG, potentially degrading performance and even causing hallucinated outputs. While various methods have been proposed to filter out such noise, we argue that identifying irrelevant information from retrieved content is inherently difficult and limited number of transformer layers can hardly solve this. Consequently, retrievers fail to filter out irrelevant documents entirely. Therefore, LLMs must be robust against such noise, but we demonstrate that standard fine-tuning approaches are often ineffective in enabling the model to selectively utilize relevant information while ignoring irrelevant content due to the structural constraints of attention patterns. To address this, we propose a novel fine-tuning method designed to enhance the model's ability to distinguish between relevant and irrelevant information within retrieved documents. Extensive experiments across multiple benchmarks show that our approach significantly improves the robustness and performance of LLMs.

Tackling the Inherent Difficulty of Noise Filtering in RAG

TL;DR

This work addresses the problem that noisy retrieved documents in retrieval-augmented generation can degrade LLM performance and induce hallucinations. It argues that filtering irrelevant content is inherently difficult for limited-capacity retrievers and that linear fine-tuning distorts attention patterns; to overcome this, the authors introduce a nonlinear attention rectification mechanism that adds a custom g(·) function to the attention update, enabling aggressive noise suppression while preserving useful reasoning. The approach is analyzed theoretically (the triple-wise nature of relevance) and validated empirically across five benchmarks, using LoRA fine-tuning and a query-before-document arrangement; results show robust improvements in noise robustness and overall accuracy. The work offers a practical path to strengthen RAG systems in real-world scenarios, by decoupling noise filtering from semantic reasoning through a carefully designed nonlinear attention update, and it highlights the potential benefits and limitations of tuning pre-trained LLMs rather than training from scratch.

Abstract

Retrieval-Augmented Generation (RAG) has become a widely adopted approach to enhance Large Language Models (LLMs) by incorporating external knowledge and reducing hallucinations. However, noisy or irrelevant documents are often introduced during RAG, potentially degrading performance and even causing hallucinated outputs. While various methods have been proposed to filter out such noise, we argue that identifying irrelevant information from retrieved content is inherently difficult and limited number of transformer layers can hardly solve this. Consequently, retrievers fail to filter out irrelevant documents entirely. Therefore, LLMs must be robust against such noise, but we demonstrate that standard fine-tuning approaches are often ineffective in enabling the model to selectively utilize relevant information while ignoring irrelevant content due to the structural constraints of attention patterns. To address this, we propose a novel fine-tuning method designed to enhance the model's ability to distinguish between relevant and irrelevant information within retrieved documents. Extensive experiments across multiple benchmarks show that our approach significantly improves the robustness and performance of LLMs.
Paper Structure (21 sections, 5 theorems, 34 equations, 6 figures, 5 tables)

This paper contains 21 sections, 5 theorems, 34 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Triple-Wise Problem
  4. Robustness of LLM When Faced with Noise
  5. Fine-tuning for Noise Filtering
  6. Attention Rectification
  7. The Auto-Regressive Nature may Cause Problem
  8. Experiments
  9. Datasets and Metrics
  10. Implementation Details
  11. Main Results
  12. The Difference of Attention Score
  13. Ablation Study with Different Activation
  14. Conclusion
  15. Experiments
  16. ...and 6 more sections

Key Result

Theorem 4.1

if there exists $\epsilon$ approximates $\widehat{attn}({\bm{x}}_i,{\bm{x}}_j)$i.e., where $\sigma$ represents softmax, then we need where $\xi_{r}=\max({\bm{x}}_i^T \Delta {\bm{W}} {\bm{x}}_j)-\min ({\bm{x}}_i^T \Delta {\bm{W}} {\bm{x}}_k),$ and $x_{j,k}$ is relevant tokens.

Figures (6)

  • Figure 1: Judging the relevance of the token 'exhausted' actually requires checking the whole meaning of the sentence, so it can hardly be done by limited number of transformer layers.
  • Figure 2: plot of $5\text{tanh}(x)$, this shows that, by tanh, we can effectively enlarge the margin between relevant and irrelevant tokens and maintain similar attention weight to those relevant ones.
  • Figure 3: The difference of attention score on answer tokens (mean(attn(answer))-mean(attn(other)), for clarity, we scale this gap by a factor of 1,000.
  • Figure 4: The performance when we set different activation function, $sum (\xi \cdot tanh(x),x)$ (sum), $\xi \cdot tanh(x)$ (tanh). We also show the performance when we use $g(x)=max/min(x,2 \xi \cdot (sigmoid(x)-0.5))$ (sigmoid). $\xi$ is set to 3, and ours means our method.
  • Figure 5: The margin of attention scores. We calculate the margin as the difference between the 90th and 10th percentile attention scores to reduce the influence of outliers
  • ...and 1 more figures

Theorems & Definitions (6)

  • Theorem 4.1
  • Theorem B.1
  • Conjecture B.2
  • Theorem B.3
  • Theorem B.4: Theorem 2 of IB_generalization
  • Theorem B.5