Representational Stability of Truth in Large Language Models

TL;DR

The paper introduces representational stability as a diagnostic for how LLMs internally encode truth, falsity, and indeterminacy. It combines a linear probing framework (sAwMIL) with controlled label perturbations to assess the robustness of veracity representations across three factual domains and sixteen open-source LLMs. Key findings show tight True/False clustering, with Neither (Fictional and Synthetic) occupying distinct regions; unfamiliar Synthetic content causes the largest boundary rotations and the highest rate of label flips, revealing epistemic brittleness tied to familiarity rather than linguistic form. The study provides a principled auditing approach that complements output accuracy, paving the way for training objectives and data curation aimed at preserving coherent, epistemically stable truth representations in LLMs.

Abstract

Large language models (LLMs) are widely used for factual tasks such as "What treats asthma?" or "What is the capital of Latvia?". However, it remains unclear how stably LLMs encode distinctions between true, false, and neither-true-nor-false content in their internal probabilistic representations. We introduce representational stability as the robustness of an LLM's veracity representations to perturbations in the operational definition of truth. We assess representational stability by (i) training a linear probe on an LLM's activations to separate true from not-true statements and (ii) measuring how its learned decision boundary shifts under controlled label changes. Using activations from sixteen open-source models and three factual domains, we compare two types of neither statements. The first are fact-like assertions about entities we believe to be absent from any training data. We call these unfamiliar neither statements. The second are nonfactual claims drawn from well-known fictional contexts. We call these familiar neither statements. The unfamiliar statements induce the largest boundary shifts, producing up to $40\%$ flipped truth judgements in fragile domains (such as word definitions), while familiar fictional statements remain more coherently clustered and yield smaller changes ($\leq 8.2\%$). These results suggest that representational stability stems more from epistemic familiarity than from linguistic form. More broadly, our approach provides a diagnostic for auditing and training LLMs to preserve coherent truth assignments under semantic uncertainty, rather than optimizing for output accuracy alone.

Figures (24)

• Figure 1: Overview of Representational Stability Evaluation. A toy example demonstrating how we assess representational stability by (a) training a True vs. Not True probe on LLM activations with True (blue), False (green) and Neither (purple) veracity values and (b) retraining the probe with perturbed labels (i.e., redefining the operational definition of truth to include the Neither statements). We compare the similarity between the original (solid) and perturbed (dashed) decision boundaries and identify how many True statements flip to Not True after the perturbation (epistemic retractions) or, conversely, how many Not True statements flip to True (epistemic expansions). Stable veracity representations should have well-clustered activations that minimize the number of epistemic retractions and expansions.
• Figure 2: Character bigram distributions of statements. Rank–frequency plots of normalized character bigram counts for True (green), False (red), Synthetic (yellow), and Fictional (blue) statements in the (a) City Locations, (b) Medical Indications, and (c) Word Definitions datasets. For each dataset, we compute per-type bigram frequencies, normalize within type, sort bigrams by their frequency under True statements, and plot log-normalized frequency with a moving-average smoothing. Across datasets, the True, False, and Synthetic distributions are nearly indistinguishable, whereas the Fictional distribution decays more slowly, marking it as structurally distinct.
• Figure 3: Average Wasserstein distance between activations. Pairwise Wasserstein distances between activation distributions of True, False, Synthetic, Fictional, and Noise statements, averaged over sixteen LLMs for the (a) City Locations, (b) Medical Indications, and (c) Word Definitions datasets. Across datasets, Synthetic activations lie closest to the True and False activations, while Fictional and Noise activations are farther from all others, indicating that unseen but fact-like statements (Synthetic) resemble factual structure, whereas Fictional statements form distinct representational clusters.
• Figure 4: Changes in the probe decision boundary under perturbations. Cosine similarity (left column) and bias difference (right column) between the baseline True vs. Not True probe and probes retrained under label perturbations for the (a,b) City Locations, (c,d) Medical Indications, and (e,f) Word Definitions datasets. Each heatmap shows results for sixteen LLMs (columns) and five perturbation conditions (rows). LLMs with leading underscores are chat models, while those without are base models. Higher cosine similarity indicates smaller rotations of the learned decision boundary, while bias difference reflects shifts in intercept. Across datasets, probes retrained with the Synthetic perturbation show the largest deviation from the original, particularly in cosine similarity.
• Figure 5: Stability of probe predictions under label perturbations for City Locations data. Bar plots show, for each of the sixteen LLMs (x-axis), how often the sAwMIL probe’s predicted label changes when retrained under four perturbations: (a)Synthetic, (b)Fictional, (c)Fictional(T), and (d)Noise. Green bars indicate True to Not True flips, while purple bars indicate Not True to True flips. The left y-axis reports the number of statements with flipped predictions, and the right y-axis reports the corresponding proportions. The Synthetic perturbation leads to the most instability, and the base models exhibit more True to Not True flips than the chat models.