ReFilter: Improving Robustness of Retrieval-Augmented Generation via Gated Filter

TL;DR

This work tackles the robustness and efficiency bottlenecks of retrieval-augmented generation when many retrieved items are noisy. It introduces ReFilter, a latent-based fusion module that performs token-level filtering via a context encoder, a gated filter, and a token fusion mechanism that injects weighted token signals into a specific LLM layer. By combining a dynamic per-token gate with a learnable position prior and a late fusion strategy, ReFilter maintains high QA performance across in-domain and out-of-domain tasks, including zero-shot biomedical QA, while reducing reliance on long context prompts. Empirical results show consistent improvements over strong baselines across multiple backbones, with favorable efficiency and robustness to retrieval noise, making ReFilter a practical solution for scalable RAG systems.

Abstract

Retrieval-augmented generation (RAG) has become a dominant paradigm for grounding large language models (LLMs) with external evidence in knowledge-intensive question answering. A core design choice is how to fuse retrieved samples into the LLMs, where existing internal fusion approaches broadly fall into query-based fusion, parametric fusion, and latent-based fusion. Despite their effectiveness at modest retrieval scales, these methods often fail to scale gracefully as the number of retrieved candidates k increases: Larger k improves evidence coverage, yet realistic top-k retrieval inevitably contains irrelevant or redundant content and increases the inference cost. To address these limitations, we propose ReFilter, a novel latent-based fusion framework that performs token-level filtering and fusion. ReFilter consists of three key components: a context encoder for encoding context features, a gated filter for weighting each token, and a token fusion module for integrating the weighted token feature into the LLM's hidden states. Our experiments across four general-domain QA benchmarks show that ReFilter consistently achieves the best average performance under both in-domain adaptation and out-of-domain transfer. ReFilter further generalizes to five biomedical QA benchmarks in zero-shot transfer without domain fine-tuning, reaching 70.01% average accuracy with Qwen2.5-14B-Instruct.

Figures (7)

• Figure 1: Three retrieval fusion paradigms in RAG: query-based fusion, parametric fusion, and latent-based fusion.
• Figure 2: The top-k recall and performance based on different backbones.
• Figure 3: Overview of our ReFilter.
• Figure 4: Robustness to top-k.
• Figure 5: Robustness to retrieval noise.