LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblance

TL;DR

This work interrogates whether LLM brittleness stems from fragile knowledge representations or merely surface-form sensitivity. It jointly evaluates three truthfulness probes, four model families, and four datasets under a suite of semantically-preserving, out-of-distribution perturbations, quantifying robustness via the relationship between probe performance and OOD-ness measured by perplexity $\mathrm{PPL}$. Across transformations, models, and domains, truth-separability degrades as inputs drift away from pre-training distributions, with negative slopes $\beta$ in the accuracy-versus-perplexity relation signaling brittle representations. The findings challenge the notion that benchmark performance reflects robust, generalizable knowledge and call for new methods to strengthen the internal representations governing truth judgments. Overall, the work highlights a fundamental limitation in current LLM knowledge encoding and provides a framework for evaluating and improving the robustness of factual representations in neural language models.

Abstract

For Large Language Models (LLMs) to be reliable, they must learn robust knowledge that can be generally applied in diverse settings -- often unlike those seen during training. Yet, extensive research has shown that LLM performance can be brittle, with models exhibiting excessive sensitivity to trivial input variations. In this work, we explore whether this brittleness is a direct result of unstable internal knowledge representations. To explore this question, we build on previous work showing that LLM representations encode statement truthfulness -- i.e., true, factual statements can be easily separated from false, inaccurate ones. Specifically, we test the robustness of learned knowledge by evaluating representation separability on samples that have undergone superficial transformations to drive them out-of-distribution (OOD), such as typos or reformulations. By applying semantically-preserving perturbations, we study how separability degrades as statements become more OOD, across four LLM families, five evaluation datasets, and three knowledge probing methods. Our results reveal that internal representations of statement truthfulness collapse as the samples' presentations become less similar to those seen during pre-training. While LLMs can often distinguish between true and false statements when they closely resemble the pre-training data, this ability is highly dependent on the statement's exact surface form. These findings offer a possible explanation for brittle benchmark performance: LLMs may learn shallow, non-robust knowledge representations that allow for only limited generalizability. Our work presents a fundamental challenge for the utility of truthfulness probes, and more broadly, calls for further research on improving the robustness of learned knowledge representations.

Figures (13)

• Figure 1: LLM truth representations degrade under superficial changes. (a) We apply semantically-preserving transformations to shift statements OOD, collapsing the representations of true and false statements. (b) We quantify robustness as the relationship between separability and OOD-ness. A hypothetical Robust model would have constant separability regardless of OOD-ness, while a hypothetical Brittle model would rapidly degrade. On MMLU, we observe that knowledge representations degrade with increasing perplexity.
• Figure 2: Truthfulness separability (probe AUC) against average perplexity on the True-False dataset for Llama 3.1 8B Instruct for the (a) linear, (b) non-linear, and (c) P(True) probes. Probe performance degrades as samples become more OOD across all tested probes, suggesting knowledge representations are not robust.
• Figure 3: Non-linear probe performance (AUC) against average perplexity for Llama 3.1 8B Instruct on (a) MMLU, (b) OpenBookQA, and (c) TruthfulQA. Despite differences in AUC on the original dataset (green dots), truthfulness representations consistently degrade on all datasets.
• Figure 4: (a) Non-linear probe performance (AUC) against average perplexity for various model families. (b) Degradation slope for the non-linear and P(True) probes at increasing Llama 3.1 Instruct scales. All models suffer degraded representations under OOD shift; increasing scale may worsen representation robustness.
• Figure 5: (a) Probe performance (AUC) for Llama 3.1 8B Instruct on the correct-only MMLU subset (red) and the full dataset (blue). (b) Non-linear probe and (c) P(True) performance (AUC) against average perplexity for correct and full sets. Knowledge representations still degrade even when the model responds correctly during benchmarking.