Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Whose Facts Win? LLM Source Preferences under Knowledge Conflicts

Jakob Schuster, Vagrant Gautam, Katja Markert

TL;DR

This work investigates how LLMs resolve knowledge conflicts when retrieved information from different sources clashes in retrieval-augmented generation. Using a synthetic, highly controlled setting across 13 open-weight models, the authors reveal a clear source credibility hierarchy—government and newspaper sources are generally preferred over individuals or social media. However, repetition bias can override these preferences, underscoring vulnerability to disinformation; a novel teacher-student fine-tuning approach combined with credibility prompting can dramatically reduce repetition bias while preserving most of the original source preferences. The study provides data and code to spur further research on credibility in knowledge-intensive NLP and has implications for designing more trustworthy RAG systems and information ecosystems.

Abstract

As large language models (LLMs) are more frequently used in retrieval-augmented generation pipelines, it is increasingly relevant to study their behavior under knowledge conflicts. Thus far, the role of the source of the retrieved information has gone unexamined. We address this gap with a novel framework to investigate how source preferences affect LLM resolution of inter-context knowledge conflicts in English, motivated by interdisciplinary research on credibility. With a comprehensive, tightly-controlled evaluation of 13 open-weight LLMs, we find that LLMs prefer institutionally-corroborated information (e.g., government or newspaper sources) over information from people and social media. However, these source preferences can be reversed by simply repeating information from less credible sources. To mitigate repetition effects and maintain consistent preferences, we propose a novel method that reduces repetition bias by up to 99.8%, while also maintaining at least 88.8% of original preferences. We release all data and code to encourage future work on credibility and source preferences in knowledge-intensive NLP.

Whose Facts Win? LLM Source Preferences under Knowledge Conflicts

TL;DR

This work investigates how LLMs resolve knowledge conflicts when retrieved information from different sources clashes in retrieval-augmented generation. Using a synthetic, highly controlled setting across 13 open-weight models, the authors reveal a clear source credibility hierarchy—government and newspaper sources are generally preferred over individuals or social media. However, repetition bias can override these preferences, underscoring vulnerability to disinformation; a novel teacher-student fine-tuning approach combined with credibility prompting can dramatically reduce repetition bias while preserving most of the original source preferences. The study provides data and code to spur further research on credibility in knowledge-intensive NLP and has implications for designing more trustworthy RAG systems and information ecosystems.

Abstract

As large language models (LLMs) are more frequently used in retrieval-augmented generation pipelines, it is increasingly relevant to study their behavior under knowledge conflicts. Thus far, the role of the source of the retrieved information has gone unexamined. We address this gap with a novel framework to investigate how source preferences affect LLM resolution of inter-context knowledge conflicts in English, motivated by interdisciplinary research on credibility. With a comprehensive, tightly-controlled evaluation of 13 open-weight LLMs, we find that LLMs prefer institutionally-corroborated information (e.g., government or newspaper sources) over information from people and social media. However, these source preferences can be reversed by simply repeating information from less credible sources. To mitigate repetition effects and maintain consistent preferences, we propose a novel method that reduces repetition bias by up to 99.8%, while also maintaining at least 88.8% of original preferences. We release all data and code to encourage future work on credibility and source preferences in knowledge-intensive NLP.
Paper Structure (77 sections, 3 equations, 32 figures, 3 tables)

This paper contains 77 sections, 3 equations, 32 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background: Credibility
  3. Data and Methodology
  4. Plausible Knowledge Conflict Pairs
  5. Synthetic Sources
  6. Newspaper.
  7. Government.
  8. Social media users.
  9. Person.
  10. Evaluation Method
  11. Models.
  12. Forced-choice prompting.
  13. Source preference metric.
  14. Significance testing.
  15. LLM Source Preferences
  16. ...and 62 more sections

Figures (32)

  • Figure 1: Source credibility hierarchy induced by evaluating $13$ LLMs on source and knowledge conflicts. However, repeating information can flip preferences.
  • Figure 2: We measure the influence of source credibility on a model's output by observing how answer probabilities for conflicting information shift when attributed to a particular source group.
  • Figure 3: Source preferences when comparing attributed and non-attributed information: All models significantly prefer attributed information. Legend in §\ref{['sec:setup-evaluation']}.
  • Figure 4: Model preferences between source types under knowledge conflicts: LLMs show strictly transitive preferences, aligning with an overall hierarchy of government $>$ newspaper $>$ individuals. Legend in §\ref{['sec:setup-evaluation']}.
  • Figure 5: Preferences when conflicting information is attributed to sources of different reputed credibility (popularity): Models prefer popular sources. Legend in §\ref{['sec:setup-evaluation']}.
  • ...and 27 more figures