Quantifying the Robustness of Retrieval-Augmented Language Models Against Spurious Features in Grounding Data

TL;DR

This work formalizes spurious features in Retrieval-Augmented Generation (RAG) and shows that semantic-agnostic cues in grounding data can bias LLM-driven readers. It introduces the SURE framework to automate spurious feature injection, preserve causal content via bidirectional entailment, and evaluate robustness with instance-level metrics (RR, WR, LR). A five-type taxonomy of perturbations and a synthetic dataset enable controlled, scalable analysis, further distilled into the SIG benchmark for efficient testing. Across multiple models and scales, the study finds that spurious features are pervasive and that robustness cannot be solved by scaling alone, highlighting directions for future defense and benchmark design.

Abstract

Robustness has become a critical attribute for the deployment of RAG systems in real-world applications. Existing research focuses on robustness to explicit noise (e.g., document semantics) but overlooks spurious features (a.k.a. implicit noise). While previous works have explored spurious features in LLMs, they are limited to specific features (e.g., formats) and narrow scenarios (e.g., ICL). In this work, we statistically confirm the presence of spurious features in the RAG paradigm, a robustness problem caused by the sensitivity of LLMs to semantic-agnostic features. Moreover, we provide a comprehensive taxonomy of spurious features and empirically quantify their impact through controlled experiments. Further analysis reveals that not all spurious features are harmful and they can even be beneficial sometimes. Extensive evaluation results across multiple LLMs suggest that spurious features are a widespread and challenging problem in the field of RAG. The code and dataset will be released to facilitate future research. We release all codes and data at: $\\\href{https://github.com/maybenotime/RAG-SpuriousFeatures}{https://github.com/maybenotime/RAG-SpuriousFeatures}$.

Figures (11)

• Figure 1: An example from the SURE framework (Sec. \ref{['sec:framework']}), illustrating the sensitivity of RAG systems to spurious features within grounding data. MD represents the Markdown format. The original retrieved document is fed into the LLMs in different formats, leading to inconsistent responses.
• Figure 2: Overview of our SURE framework. We provide a Comprehensive Taxonomy that includes five types of spurious features, further divided into 13 subtypes of perturbations (left section). To construct the testbed, we prepare raw instances initially and then synthesize the modified instances through a workflow consisting of Spurious Features Injection and Causal Features Preservation (middle section). By applying carefully tailored metrics for Robustness Evaluation, we quantify the robustness of target RAG systems (right section).
• Figure 3: A comparison of dataset-level metric (Acc) and instance-level metric (RR) for robustness evaluation. ✔ and ✘ indicate the correctness of responses. In this example, the value of RR reflects the unrobustness at the instance level, while dataset-level metric overlook the sensitivity of RALMs to spurious features within documents.
• Figure 4: Comparison of Robustness rates across six SOTA LLMs.
• Figure 5: Scaling analysis of robustness to spurious features.