Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?

TL;DR

The paper investigates why language model outputs diverge from the model's internal truth representations and proposes a taxonomy of disagreements—confabulation, deception, and heterogeneity. It compares two evaluation pathways, direct querying and hidden-state probing with linear classifiers, across BoolQ, SciQ, and CREAK using GPT-2-XL and GPT-J, and demonstrates that probes generally offer better calibration and accuracy, while ensembling probes and queries yields additional gains in several datasets. The findings argue that most mismatches arise from multiple non-deceptive prediction pathways rather than an explicit intent to lie, and that combining pathways can enhance factual verification for downstream tasks. These insights inform how to interpret LM truthfulness, guide prompt design, and support safer deployment by highlighting when and how to blend inference pathways to improve reliability.

Abstract

Neural language models (LMs) can be used to evaluate the truth of factual statements in two ways: they can be either queried for statement probabilities, or probed for internal representations of truthfulness. Past work has found that these two procedures sometimes disagree, and that probes tend to be more accurate than LM outputs. This has led some researchers to conclude that LMs "lie" or otherwise encode non-cooperative communicative intents. Is this an accurate description of today's LMs, or can query-probe disagreement arise in other ways? We identify three different classes of disagreement, which we term confabulation, deception, and heterogeneity. In many cases, the superiority of probes is simply attributable to better calibration on uncertain answers rather than a greater fraction of correct, high-confidence answers. In some cases, queries and probes perform better on different subsets of inputs, and accuracy can further be improved by ensembling the two. Code is available at github.com/lingo-mit/lm-truthfulness.

Figures (4)

• Figure 1: Varieties of disagreement between language model outputs and internal states. We evaluate two approaches for answering questions or verifying statements: querying models directly for answers (A), or training a binary classifier to probe their internal states (B). Probes and queries sometimes disagree. We propose a taxonomy of different query--probe disagreement types (C); across several models and datasets, we find that disagreements mostly occur in situations where either probes or queries are uncertain (D).
• Figure 2: Calibration of GPT-J queries and probes. Each point represents a group of predictions: the horizontal axis shows the query's average confidence $\mathbb{E}_{q, a} p(a \mid q)$, the vertical axis shows the query's empirical accuracy $\mathbb{E} [a \textrm{ is correct}]$, and point radius shows the number of predictions in the group. Probes are substantially better calibrated than LM queries.
• Figure 3: Distribution of query and probe predictions. "Deception"-like results (orange) do not feature particularly prominently compared to other outcomes. Note that heatmaps (left) use the same shade of gray for "both correct" and "both incorrect", even though they are distinguished in histograms (right) to enable direct comparison.
• Figure 4: Sparse probing results. In these experiments, we train the same probes as in the main paper, but with a varying $\ell_1$ penalty applied to the probing objective to encourage the discovery of sparse solutions. Here we report the level of sparsity, probe accuracy, and distribution of disagreement types as we vary the strength of the regularizer within $\{0, 0.01, 0.03, 0.1\}$. Except at extremely high sparsity values, both accuracies and error distributions remain similar to those reported in the main set of experiments.