Out of Style: RAG's Fragility to Linguistic Variation

TL;DR

This paper conducts the first large-scale, end-to-end analysis of how linguistic variation in user queries affects Retrieval-augmented Generation (RAG) systems. By rewriting queries along formality, readability, politeness, and grammatical correctness across four QA datasets and two retrievers with nine LLMs, it reveals substantial robustness gaps in both retrieval and generation stages and demonstrates cascading error propagation within the RAG pipeline. The study shows that larger LLM scales do not reliably mitigate these gaps and that advanced retrieval techniques (e.g., HyDE, reranking) offer limited resilience gains. The findings underscore the need for robust, end-to-end design principles and techniques to ensure reliable performance for diverse, real-world user interactions. Practically, the work informs future RAG development toward more robust retrieval, context utilization, and error-tolerant generation under linguistically varied inputs.

Abstract

Despite the impressive performance of Retrieval-augmented Generation (RAG) systems across various NLP benchmarks, their robustness in handling real-world user-LLM interaction queries remains largely underexplored. This presents a critical gap for practical deployment, where user queries exhibit greater linguistic variations and can trigger cascading errors across interdependent RAG components. In this work, we systematically analyze how varying four linguistic dimensions (formality, readability, politeness, and grammatical correctness) impact RAG performance. We evaluate two retrieval models and nine LLMs, ranging from 3 to 72 billion parameters, across four information-seeking Question Answering (QA) datasets. Our results reveal that linguistic reformulations significantly impact both retrieval and generation stages, leading to a relative performance drop of up to 40.41% in Recall@5 scores for less formal queries and 38.86% in answer match scores for queries containing grammatical errors. Notably, RAG systems exhibit greater sensitivity to such variations compared to LLM-only generations, highlighting their vulnerability to error propagation due to linguistic shifts. These findings highlight the need for improved robustness techniques to enhance reliability in diverse user interactions.

Figures (5)

• Figure 1: RAG systems demonstrate overall performance degradation when queries are rewritten to be less formal, more polite, less readable, and with grammatical errors. For the traditional NLP query (left), the RAG systems successfully retrieve related information and generate the correct answer, while the less formal queries (right) retrieve incorrect information. The linguistic variation on formality causes significant performance drops: 40.41% decrease in Recall@5 and 33.38% decrease in answer match (AM) score on the MS MARCO dataset.
• Figure 2: PopQA and Natural Questions (NQ) LLMs scaling results, augmented with ModernBERT, Contriever, and LLM-only generation (Vanilla). Retrieval-augmented generation is more sensitive to linguistic variations than the LLM-only generation.
• Figure 3: Retrieval (ModernBERT, R@5 Score) and generation (Qwen2.5-7B-Instruct, AM Score) performance across different RAG settings on PopQA. We find that (1) adding HyDE and Rerank to the RAG pipeline improves the robustness to linguistic variations, but still lags behind original queries in performance. (2) HyDE improves robustness but slightly reduces performance on original queries. (3) Reranking improves performance on both original and rewritten queries.
• Figure 4: Average Recall@K increase as Number of Top-K Documents increases – Contriever.
• Figure 5: Average Recall@K increase as Number of Top-K Documents increases – ModernBERT.