The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriors

TL;DR

This work investigates how prompt-conditioned beliefs are encoded and updated in large language models by examining a controlled Gaussian-time-series task with prompts drawn from $\mathcal{N}_{\mu,\sigma}$. The authors reveal curved belief manifolds $\mathcal{M}_{\mu,\sigma}$ in activation space and introduce Linear Field Probes (LFP) to tile these manifolds, enabling geometry-aware interventions. They show that belief dynamics involve rapid mean adaptation and slower variance equilibration when distributions switch, and demonstrate that linear steering along a manifold, rather than along a fixed direction, better preserves the target belief family. The findings argue for a geometric view of latent beliefs in LLMs, with practical implications for interpretable probing and reliable belief editing using dual-space field geometry.

Abstract

Large language models (LLMs) represent prompt-conditioned beliefs (posteriors over answers and claims), but we lack a mechanistic account of how these beliefs are encoded in representation space, how they update with new evidence, and how interventions reshape them. We study a controlled setting in which Llama-3.2 generates samples from a normal distribution by implicitly inferring its parameters (mean and standard deviation) given only samples from the distribution in context. We find representations of curved "belief manifolds" for these parameters form with sufficient in-context learning and study how the model adapts when the distribution suddenly changes. While standard linear steering often pushes the model off-manifold and induces coupled, out-of-distribution shifts, geometry and field-aware steering better preserves the intended belief family. Our work demonstrates an example of linear field probing (LFP) as a simple approach to tile the data manifold and make interventions that respect the underlying geometry. We conclude that rich structure emerges naturally in LLMs and that purely linear concept representations are often an inadequate abstraction.

Figures (12)

• Figure 1: The Shape of beliefs. (A) Stochastic time series $u(t) \in \llbracket 0, 999 \rrbracket$ are generated from normal distributions $\mathcal{N}_{\mu,\sigma}$ and passed into Llama 3.2 as strings. (B) Representations (PCA at layer 14) form curved manifolds parametrized by $(\mu, \sigma_0=100)$ (pink to green) and $(\mu_0=500,\sigma)$ (shades of purple). They encode the model's current posterior inferred from the input data. (C) Softmax outputs, shown here in as a 3D embedding representation from inPCA, mirror the geometry of activations. (D) Softmax probabilities, alternatively represented by their projection onto the subset of tokens corresponding to integers from 0 to 999, closely reflect the input distributions.
• Figure 2: Belief dynamics. Input (A) is a time series with a sharp transition between two distinct regimes, $\mathcal{N}_{300,\sigma_0} \rightarrow \mathcal{N}_{700,\sigma_0}$ at $t=1000$. Output (B) shows the model quickly adapting its mean after the switch while broadening its variance, which relaxes back to its true value after about 300 tokens. This switch in belief is apparent in the model's activations (C). The trajectory in probability space (D) shows two attractors corresponding to the true input distributions, and the path taken by the model across them.
• Figure 3: Linear field probes. (A) Separability: probe accuracy (on test set) on all $\mu$-indexed representations, for all layer. (B) Continuity: cosine similarity between probes, showing smooth variation over $\mu$ and revealing structured geometry over the domain. (C) Interpolation: cosine similarity between true and (kernel-) interpolated (see \ref{['sec:probe_interpolation']}) vectors at intermediate $\mu$; for reference, cosine similarity between two random vectors in a space of $d=2048$ dimensions has mean 0 and standard deviation $1/\sqrt{2048} \simeq 0.02$. (D) Transfer: probes only transfer locally; here showing a probe trained on $\mu = \{ 300, 350 \}$ (layer 0) and applied on sets $\mu = \{ 350, 400 \} (\Delta\mu =50)$, $\mu = \{ 400, 450 \} (\Delta\mu = 100)$, etc. (An untrained binary probe has accuracy 0.5.)
• Figure 4: Field geometry. (Left) Cumulative variance explained as a function of rank of the eigenvalues of the LFP Gram matrices (Fig \ref{['fig:LFP']}B), for each layer. The intrinsic dimensionality increases with layers, but drops at the last layer (L15). (Right) Kernel PCA embedding of the first 3 eigenvectors of the Gram matrix at layer 15. This represents the field geometry, dual to the activation space manifold.
• Figure 5: Steering based on activation geometry: linear vs manifold-aware. The manifold of activations (A) for $\mu \in [300, 700]$ provides a principled basis for steering directions, for example a centroid-to-centroid vector (orange), and a manifold-fitting spline (yellow). The corresponding resulting logits are shown in (B). Steering linearly brings steered logits far out of the $\sigma=100$-manifold (orange path), while steering along the manifold keeps outputs significantly more aligned with the targeted behavior.