How Context Shapes Truth: Geometric Transformations of Statement-level Truth Representations in LLMs

TL;DR

The paper addresses how context reshapes statement-level truth representations in LLMs by measuring geometric transforms of truth vectors in the residual stream. Using four instruction-tuned models and four diverse datasets, it computes the directional change $\theta$ and relative magnitude $rm$ between truth vectors with and without context across layers. The key findings reveal a three-phase pattern of directional change, general amplification of truth-vector separation by context, and variable sensitivity to context relevance across model scales, with stronger effects for context that conflicts with parametric knowledge. This geometric characterization informs the design of retrieval-augmented and in-context learning systems and contributes to interpretability of how contextual knowledge is integrated in LLMs.

Abstract

Large Language Models (LLMs) often encode whether a statement is true as a vector in their residual stream activations. These vectors, also known as truth vectors, have been studied in prior work, however how they change when context is introduced remains unexplored. We study this question by measuring (1) the directional change ($θ$) between the truth vectors with and without context and (2) the relative magnitude of the truth vectors upon adding context. Across four LLMs and four datasets, we find that (1) truth vectors are roughly orthogonal in early layers, converge in middle layers, and may stabilize or continue increasing in later layers; (2) adding context generally increases the truth vector magnitude, i.e., the separation between true and false representations in the activation space is amplified; (3) larger models distinguish relevant from irrelevant context mainly through directional change ($θ$), while smaller models show this distinction through magnitude differences. We also find that context conflicting with parametric knowledge produces larger geometric changes than parametrically aligned context. To the best of our knowledge, this is the first work that provides a geometric characterization of how context transforms the truth vector in the activation space of LLMs.

Figures (12)

• Figure 1: Overview of our approach(1) For a statement $k$, we generate 4 inputs by varying the [Selected Choice] and presence of context. The LLM is instructed to generate the completion based on the [Selected Choice]. (2) We extract the residual stream activations for generating the first token and label them as true or false based on the ground truth. (3) We compare the truth vectors with and without context ($v_{k,nc}$ and $v_{k,c}$), calculating directional change $\theta$ and relative magnitudinal change $\frac{||v_{k,c}||^2}{||v_{k,nc}||^2}$ across all the layers.
• Figure 2: (a) Prompt to generate completion supporting the statement with context. (b) Given a statement, we create four prompts: supporting or refuting the statement, each with or without context. The LLM generates a completion for each prompt.
• Figure 3: For a statement $k$, $v_{k,nc}$ (AB) is the truth vector without context and $v_{k,c}$ (CD) is the truth vector when context is added. $\theta$ is the angle between $v_{k,nc}$ and $v_{k,c}$ denoting the directional change (Eq. \ref{['eq:theta']}). To track relative magnitudes, we compute the ratio of $L_2$ distances: $\frac{CD}{AB}$, $\frac{AD}{AB}$ and $\frac{BC}{AB}$ as per Eq. \ref{['eq:rm1']}, \ref{['eq:rm2']} & \ref{['eq:rm3']}.
• Figure 4: Layer wise plot of average $\theta$ in degrees across different models and datasets indicating the directional change in truth vectors when context is added. Vertical red lines indicate the beginning of a new phase. Across all the settings, we observe three phases: Phase-1, where the truth vectors are almost orthogonal, Phase-2, where the truth vectors become more similar and finally Phase-3, where truth vectors stabilize or continue increasing.
• Figure 5: Layer wise plot of average relative magnitudes across different models and datasets indicating the increase in the magnitude of truth vector when context is added. Early layers show variability, followed by a peak in the middle layers. The values decrease and stabilize towards the final layers.