Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Knowledge Drift in LLMs through Misinformation

Alina Fastowski, Gjergji Kasneci

TL;DR

This work analyzes the susceptibility of state-of-the-art LLMs to factual inaccuracies when they encounter false information in a QnA scenario, an issue that can lead to a phenomenon the authors refer to as *knowledge drift*, which significantly undermines the trustworthiness of these models.

Abstract

Large Language Models (LLMs) have revolutionized numerous applications, making them an integral part of our digital ecosystem. However, their reliability becomes critical, especially when these models are exposed to misinformation. We primarily analyze the susceptibility of state-of-the-art LLMs to factual inaccuracies when they encounter false information in a QnA scenario, an issue that can lead to a phenomenon we refer to as *knowledge drift*, which significantly undermines the trustworthiness of these models. We evaluate the factuality and the uncertainty of the models' responses relying on Entropy, Perplexity, and Token Probability metrics. Our experiments reveal that an LLM's uncertainty can increase up to 56.6% when the question is answered incorrectly due to the exposure to false information. At the same time, repeated exposure to the same false information can decrease the models uncertainty again (-52.8% w.r.t. the answers on the untainted prompts), potentially manipulating the underlying model's beliefs and introducing a drift from its original knowledge. These findings provide insights into LLMs' robustness and vulnerability to adversarial inputs, paving the way for developing more reliable LLM applications across various domains. The code is available at https://github.com/afastowski/knowledge_drift.

Understanding Knowledge Drift in LLMs through Misinformation

TL;DR

This work analyzes the susceptibility of state-of-the-art LLMs to factual inaccuracies when they encounter false information in a QnA scenario, an issue that can lead to a phenomenon the authors refer to as *knowledge drift*, which significantly undermines the trustworthiness of these models.

Abstract

Large Language Models (LLMs) have revolutionized numerous applications, making them an integral part of our digital ecosystem. However, their reliability becomes critical, especially when these models are exposed to misinformation. We primarily analyze the susceptibility of state-of-the-art LLMs to factual inaccuracies when they encounter false information in a QnA scenario, an issue that can lead to a phenomenon we refer to as *knowledge drift*, which significantly undermines the trustworthiness of these models. We evaluate the factuality and the uncertainty of the models' responses relying on Entropy, Perplexity, and Token Probability metrics. Our experiments reveal that an LLM's uncertainty can increase up to 56.6% when the question is answered incorrectly due to the exposure to false information. At the same time, repeated exposure to the same false information can decrease the models uncertainty again (-52.8% w.r.t. the answers on the untainted prompts), potentially manipulating the underlying model's beliefs and introducing a drift from its original knowledge. These findings provide insights into LLMs' robustness and vulnerability to adversarial inputs, paving the way for developing more reliable LLM applications across various domains. The code is available at https://github.com/afastowski/knowledge_drift.
Paper Structure (17 sections, 3 equations, 3 figures, 4 tables)

This paper contains 17 sections, 3 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Language model architectures and capabilities.
  5. Uncertainty in LLMs.
  6. LLM factual knowledge and calibration.
  7. Experiments
  8. Experimental Setup
  9. Dataset and Models.
  10. Performance evaluation.
  11. Uncertainty metrics.
  12. Baseline and information injection.
  13. Results and Discussion
  14. Evaluating Model Uncertainty and Knowledge Retention with Varying Prompt Integrity.
  15. Influence of Repeated False Information on Model Confidence and Accuracy.
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Answers produced by state-of-the-art LLMs on "What's Rambo's first name?" with no perturbation (col. 1), with false information injection (cols. 2 & 3), and with random information injection (col. 4). Green boxes indicate correct answers; red are incorrect. The transparency of the boxes indicates the uncertainty of the model: i.e., the lighter, the more uncertain. Notice how injecting the same false information multiple times makes LLMs more uncertain (see GPT-3.5) and can even shift their original correct answer to a wrong one (see Mistral and LLaMA).
  • Figure 2: Prompt V1: Changes in uncertainty when repeating the false information $\times k$.
  • Figure 3: Prompt V2: Changes in uncertainty when repeating the false information $\times k$.