Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emergence of Linear Truth Encodings in Language Models

Shauli Ravfogel, Gilad Yehudai, Tal Linzen, Joan Bruna, Alberto Bietti

TL;DR

The paper tackles how large language models develop linear subspaces that separate true from false statements. By building a transparent one-layer transformer toy model and a minimal truth-cooccurrence data generator, it shows a two-phase learning dynamic: rapid memorization of subject–attribute pairs followed by the gradual emergence of a linear truth direction that sharpens predictions for truthful contexts, aided by layer normalization. The authors corroborate the mechanism with experiments in synthetic settings, natural language data (CounterFact), and pretrained LLMs, demonstrating both a predictive latent truth variable and a practical way to steer model outputs along truth-relevant directions. This work provides a mechanistic account linking data distribution, associative memory circuits in transformers, and normalization-induced sharpening to the emergence of linear truth encodings, with implications for reducing hallucinations and improving factuality in LMs.

Abstract

Recent probing studies reveal that large language models exhibit linear subspaces that separate true from false statements, yet the mechanism behind their emergence is unclear. We introduce a transparent, one-layer transformer toy model that reproduces such truth subspaces end-to-end and exposes one concrete route by which they can arise. We study one simple setting in which truth encoding can emerge: a data distribution where factual statements co-occur with other factual statements (and vice-versa), encouraging the model to learn this distinction in order to lower the LM loss on future tokens. We corroborate this pattern with experiments in pretrained language models. Finally, in the toy setting we observe a two-phase learning dynamic: networks first memorize individual factual associations in a few steps, then -- over a longer horizon -- learn to linearly separate true from false, which in turn lowers language-modeling loss. Together, these results provide both a mechanistic demonstration and an empirical motivation for how and why linear truth representations can emerge in language models.

Emergence of Linear Truth Encodings in Language Models

TL;DR

The paper tackles how large language models develop linear subspaces that separate true from false statements. By building a transparent one-layer transformer toy model and a minimal truth-cooccurrence data generator, it shows a two-phase learning dynamic: rapid memorization of subject–attribute pairs followed by the gradual emergence of a linear truth direction that sharpens predictions for truthful contexts, aided by layer normalization. The authors corroborate the mechanism with experiments in synthetic settings, natural language data (CounterFact), and pretrained LLMs, demonstrating both a predictive latent truth variable and a practical way to steer model outputs along truth-relevant directions. This work provides a mechanistic account linking data distribution, associative memory circuits in transformers, and normalization-induced sharpening to the emergence of linear truth encodings, with implications for reducing hallucinations and improving factuality in LMs.

Abstract

Recent probing studies reveal that large language models exhibit linear subspaces that separate true from false statements, yet the mechanism behind their emergence is unclear. We introduce a transparent, one-layer transformer toy model that reproduces such truth subspaces end-to-end and exposes one concrete route by which they can arise. We study one simple setting in which truth encoding can emerge: a data distribution where factual statements co-occur with other factual statements (and vice-versa), encouraging the model to learn this distinction in order to lower the LM loss on future tokens. We corroborate this pattern with experiments in pretrained language models. Finally, in the toy setting we observe a two-phase learning dynamic: networks first memorize individual factual associations in a few steps, then -- over a longer horizon -- learn to linearly separate true from false, which in turn lowers language-modeling loss. Together, these results provide both a mechanistic demonstration and an empirical motivation for how and why linear truth representations can emerge in language models.
Paper Structure (45 sections, 5 theorems, 78 equations, 16 figures, 1 table)

This paper contains 45 sections, 5 theorems, 78 equations, 16 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. The Truth Co-occurrence Hypothesis
  4. Data Generating Process
  5. Data format.
  6. Truth as a latent variable.
  7. Truth as a latent variable.
  8. Analysis on a Toy Model
  9. Setup.
  10. Probing the mechanism and its emergence.
  11. Structure of the value matrix $W$.
  12. Linear separation and sharpening mechanism.
  13. Theoretical analysis of training dynamics.
  14. Experiments
  15. Synthetic Setting
  16. ...and 30 more sections

Key Result

Theorem 1

Suppose we have a solution that satisfies Eqs. eq:We_x-eq:Wp_3. Denote by $c:= 2+\frac{\bar{\gamma}^2(2N-2) + 2\alpha_1^2 + \beta_1^2}{9}$ For any $x, x'$ and $y\neq g(x)$, we have:

Figures (16)

  • Figure 1: Visualization of the value matrix for the one-layer model at different training steps. We see that the $e_x \to u_{g(x)}$ block is learned first, along with the $p_t \to \bar{u}$ block. Later the $e_x \to -e_x$ and $e_y \to -u_y$ blocks, and finally the $e_y \to e_{g^{-1}(y)}$ block.
  • Figure 2: Visualization of representations on true (top) and false (bottom) sequences. The plots show representations before (left) and after (center) layer-norm, as well as predicted probabilities (right).
  • Figure 3: Truth linear classification results alongside probability assigned by the LM to the true attribute on false sequences.
  • Figure 4: LM representations over false sequences.
  • Figure 5: Truth linear classification results alongside probability assigned by the LM to the true attribute on false sequences.
  • ...and 11 more figures

Theorems & Definitions (7)

  • Theorem 1: Sharpening of $y'$ predictions
  • Theorem 2: Linear truth direction
  • Theorem 3: Sequential gradient learning; informal
  • Lemma 1
  • Theorem 3
  • proof
  • proof