Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rational Synthesizers or Heuristic Followers? Analyzing LLMs in RAG-based Question-Answering

Atharv Naphade

TL;DR

This work investigates how large language models synthesize groups of retrieved documents in Retrieval-Augmented Generation when faced with conflicting evidence. It introduces GroupQA, a dataset of 1,635 controversial questions paired with 15,058 evidence documents, designed to probe group-level dynamics such as evidence quantity, redundancy, and ordering. Key findings show that paraphrased repetition (illusory truth) often drives model decisions more than diverse evidence, and the first documents in a group (primacy effect) exert outsized influence; model scale increases belief stability while reducing plasticity. The study discusses vulnerabilities in RAG systems, notably context stuffing and attribution unfaithfulness, and suggests that improving rational synthesis requires deliberately incorporating dissenting viewpoints and debiasing aggregation mechanisms. Overall, GroupQA provides a rigorous benchmark for understanding and mitigating heuristic aggregation in RAG-based QA systems.

Abstract

Retrieval-Augmented Generation (RAG) is the prevailing paradigm for grounding Large Language Models (LLMs), yet the mechanisms governing how models integrate groups of conflicting retrieved evidence remain opaque. Does an LLM answer a certain way because the evidence is factually strong, because of a prior belief, or merely because it is repeated frequently? To answer this, we introduce GroupQA, a curated dataset of 1,635 controversial questions paired with 15,058 diversely-sourced evidence documents, annotated for stance and qualitative strength. Through controlled experiments, we characterize group-level evidence aggregation dynamics: Paraphrasing an argument can be more persuasive than providing distinct independent support; Models favor evidence presented first rather than last, and Larger models are increasingly resistant to adapt to presented evidence. Additionally, we find that LLM explanations to group-based answers are unfaithful. Together, we show that LLMs behave consistently as vulnerable heuristic followers, with direct implications for improving RAG system design.

Rational Synthesizers or Heuristic Followers? Analyzing LLMs in RAG-based Question-Answering

TL;DR

This work investigates how large language models synthesize groups of retrieved documents in Retrieval-Augmented Generation when faced with conflicting evidence. It introduces GroupQA, a dataset of 1,635 controversial questions paired with 15,058 evidence documents, designed to probe group-level dynamics such as evidence quantity, redundancy, and ordering. Key findings show that paraphrased repetition (illusory truth) often drives model decisions more than diverse evidence, and the first documents in a group (primacy effect) exert outsized influence; model scale increases belief stability while reducing plasticity. The study discusses vulnerabilities in RAG systems, notably context stuffing and attribution unfaithfulness, and suggests that improving rational synthesis requires deliberately incorporating dissenting viewpoints and debiasing aggregation mechanisms. Overall, GroupQA provides a rigorous benchmark for understanding and mitigating heuristic aggregation in RAG-based QA systems.

Abstract

Retrieval-Augmented Generation (RAG) is the prevailing paradigm for grounding Large Language Models (LLMs), yet the mechanisms governing how models integrate groups of conflicting retrieved evidence remain opaque. Does an LLM answer a certain way because the evidence is factually strong, because of a prior belief, or merely because it is repeated frequently? To answer this, we introduce GroupQA, a curated dataset of 1,635 controversial questions paired with 15,058 diversely-sourced evidence documents, annotated for stance and qualitative strength. Through controlled experiments, we characterize group-level evidence aggregation dynamics: Paraphrasing an argument can be more persuasive than providing distinct independent support; Models favor evidence presented first rather than last, and Larger models are increasingly resistant to adapt to presented evidence. Additionally, we find that LLM explanations to group-based answers are unfaithful. Together, we show that LLMs behave consistently as vulnerable heuristic followers, with direct implications for improving RAG system design.
Paper Structure (67 sections, 5 equations, 20 figures, 4 tables)

This paper contains 67 sections, 5 equations, 20 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work and Motivation
  3. Conflicting Information in RAG.
  4. Sycophancy and Contextual Persuasion.
  5. The Illusory Truth Effect in LLMs.
  6. Attention and Ordering Biases.
  7. Multi-Document Reasoning.
  8. The GroupQA Dataset
  9. Collecting contentious questions.
  10. Collecting evidence paragraphs.
  11. Filtering and Stance Labeling.
  12. Creating conflicting examples.
  13. Experimental Results
  14. Model Answers
  15. Robustness of Model Answers.
  16. ...and 52 more sections

Figures (20)

  • Figure 1: Aggregate shift in belief probability before and after CoT. The negligible delta suggests reasoning does not significantly alter the underlying belief distribution.
  • Figure 2: Scaling Trend: Model Size vs. Plasticity. Analysis of 10 open-weight models
  • Figure 3: (Llama-3.1-70B-Instruct). Paraphrased vs distinct evidence flipping rate with all documents opposed to model prior.
  • Figure 4: Conflict awareness metrics. Conflict Detection. measures the proportion of instances where the model explicitly flags a contradiction. Attribution Accuracy measures the precision of assigning the correct stance (Yes/No) to each retrieved document.
  • Figure 5: Evidence Saturation in Balanced Contexts (Llama-3.1-70B-Instruct). Comparison of flip rates between single-sided (Parametric Prior) and balanced (Conflicting Context) initialization. When the model faces conflicting information, it enters a stable state where redundant evidence provides diminishing returns.
  • ...and 15 more figures