Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals

TL;DR

RAGuard introduces a realism-focused benchmark for evaluating retrieval-augmented generation under misleading retrievals in political fact-checking. By combining PolitiFact verdicts with Reddit-derived retrieval and an LLM-guided annotation protocol, the dataset exposes how current RAG systems struggle when evidence is misframed or noisy, often performing worse than zero-shot baselines. The work provides a unified evidence taxonomy (supporting, misleading, unrelated), three evaluation tasks (Zero-Context, Standard RAG, Oracle Retrieval), and comprehensive baselines across open- and closed-source LLMs, revealing substantial robustness gaps. Human annotators consistently outperform models in this setting, highlighting the need for adversarial training and safer retrieval strategies to improve reliability in real-world misinformation contexts. This benchmark thus drives progress toward more trustworthy RAG systems that can handle real-world, conflicting information in high-stakes domains.

Abstract

Retrieval-augmented generation (RAG) has shown impressive capabilities in mitigating hallucinations in large language models (LLMs). However, LLMs struggle to maintain consistent reasoning when exposed to misleading or conflicting evidence, especially in real-world domains such as politics, where information is polarized or selectively framed. Mainstream RAG benchmarks evaluate models under clean retrieval settings, where systems generate answers from gold-standard documents, or under synthetically perturbed settings, where documents are artificially injected with noise. These assumptions fail to reflect real-world conditions, often leading to an overestimation of RAG system performance. To address this gap, we introduce RAGuard, the first benchmark to evaluate the robustness of RAG systems against misleading retrievals. Unlike prior benchmarks that rely on synthetic noise, our fact-checking dataset captures naturally occurring misinformation by constructing its retrieval corpus from Reddit discussions. It categorizes retrieved evidence into three types: supporting, misleading, and unrelated, providing a realistic and challenging testbed for assessing how well RAG systems navigate different types of evidence. Our experiments reveal that, when exposed to potentially misleading retrievals, all tested LLM-powered RAG systems perform worse than their zero-shot baselines (i.e., no retrieval at all), while human annotators consistently perform better, highlighting LLMs' susceptibility to noisy environments. To our knowledge, RAGuard is the first benchmark to systematically assess the robustness of the RAG against misleading evidence. We expect this benchmark to drive future research toward improving RAG systems beyond idealized datasets, making them more reliable for real-world applications. The dataset is available at https://huggingface.co/datasets/UCSC-IRKM/RAGuard.

Figures (10)

• Figure 1: Examples of LLM and human performance on a false claim (left) and a true claim (right) from RAGuard. While the LLM initially classified both claims correctly, it later reversed its decisions due to misleading retrieved context. In contrast, human judgments remained consistent.
• Figure 1: Comparison of RAGuard with fact-checking and noisy RAG datasets. "FC" indicates suitability for fact-checking, and "ROB" for LLM robustness evaluation. Columns reflect evaluation focus, evidence types, and dataset characteristics.
• Figure 2: Taxonomy of document types used in our benchmark (supporting, misleading, unrelated), along with types of evidence included in related datasets.
• Figure 3: Overview of RAGuard, including dataset statistics and word frequencies.
• Figure 4: RAGuard dataset construction, consisting of three stages to obtain claims and verdicts, associated documents, and labels for the each document's relationship to the claim and verdict.