Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks

Xiaodong Yu, Hao Cheng, Xiaodong Liu, Dan Roth, Jianfeng Gao

TL;DR

This work addresses the challenge of reliably evaluating retrieval-augmented LLMs, which static QA benchmarks fail to capture due to data contamination and memorization. It introduces ReEval, a framework that generates dynamic adversarial test cases by perturbing evidence through two strategies—answer swapping and context enriching—guided by prompt chaining with a pivot LLM. Experiments on Natural Questions and RealTimeQA across multiple open- and closed-book prompts reveal that perturbations induce hallucinations across models and that adversarial tests transfer between models, enabling cost-effective evaluation with a small pivot. The findings highlight the need for robust faithfulness testing in retrieval-augmented LLMs and offer a practical pathway toward diagnosing and potentially mitigating hallucinations in real-world information scenarios.

Abstract

Despite remarkable advancements in mitigating hallucinations in large language models (LLMs) by retrieval augmentation, it remains challenging to measure the reliability of LLMs using static question-answering (QA) data. Specifically, given the potential of data contamination (e.g., leading to memorization), good static benchmark performance does not ensure that model can reliably use the provided evidence for responding, which is essential to avoid hallucination when the required knowledge is new or private. Inspired by adversarial machine learning, we investigate the feasibility of automatically perturbing existing static one for dynamic evaluation. Specifically, this paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases for evaluating the LLMs' reliability in using new evidence for answering. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets on a collection of LLMs under various prompting settings. Our generated data is human-readable and useful to trigger hallucination in LLM. Accurate models on static data are observed to produce unsupported answers from the perturbed evidence, with pronounced accuracy drops across LLMs including GPT-4. We find that our adversarial examples are transferable across all considered LLMs. The examples generated by a small model can be used to evaluate a much larger model, making our approach cost-effective.

ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks

TL;DR

This work addresses the challenge of reliably evaluating retrieval-augmented LLMs, which static QA benchmarks fail to capture due to data contamination and memorization. It introduces ReEval, a framework that generates dynamic adversarial test cases by perturbing evidence through two strategies—answer swapping and context enriching—guided by prompt chaining with a pivot LLM. Experiments on Natural Questions and RealTimeQA across multiple open- and closed-book prompts reveal that perturbations induce hallucinations across models and that adversarial tests transfer between models, enabling cost-effective evaluation with a small pivot. The findings highlight the need for robust faithfulness testing in retrieval-augmented LLMs and offer a practical pathway toward diagnosing and potentially mitigating hallucinations in real-world information scenarios.

Abstract

Despite remarkable advancements in mitigating hallucinations in large language models (LLMs) by retrieval augmentation, it remains challenging to measure the reliability of LLMs using static question-answering (QA) data. Specifically, given the potential of data contamination (e.g., leading to memorization), good static benchmark performance does not ensure that model can reliably use the provided evidence for responding, which is essential to avoid hallucination when the required knowledge is new or private. Inspired by adversarial machine learning, we investigate the feasibility of automatically perturbing existing static one for dynamic evaluation. Specifically, this paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases for evaluating the LLMs' reliability in using new evidence for answering. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets on a collection of LLMs under various prompting settings. Our generated data is human-readable and useful to trigger hallucination in LLM. Accurate models on static data are observed to produce unsupported answers from the perturbed evidence, with pronounced accuracy drops across LLMs including GPT-4. We find that our adversarial examples are transferable across all considered LLMs. The examples generated by a small model can be used to evaluate a much larger model, making our approach cost-effective.
Paper Structure (25 sections, 2 figures, 15 tables)

This paper contains 25 sections, 2 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Faithfulness of Augmented LLM.
  4. Adversarial Attacks & Transferability.
  5. ReEval Framework
  6. Seed Case Selection.
  7. Evidence Perturbation.
  8. Re-evaluation.
  9. Category 1: Answer Swapping
  10. Category 2: Context Enriching
  11. Experiments
  12. Experiment Settings
  13. Evaluation Metrics.
  14. Source Data.
  15. Generated Data.
  16. ...and 10 more sections

Figures (2)

  • Figure 1: An example of how the original evidence is edited (answer swapping and context enriching) by ReEval. The question is "when did athens emerges as wealthiest greek city state?". "the early 4th century BCE" and "the late 6th century BCE"is the desirable answers for answer swapping (Category 1) and context enriching (Category 2), respectively. ChatGPT answers are next to the emoji.
  • Figure 2: The pipeline of ReEval, including identifying seed cases, generating new tests, and hallucination evaluation.