Entropy-Lens: Uncovering Decision Strategies in LLMs

TL;DR

Entropy-Lens introduces a low-dimensional, per-layer entropy profile to summarize token-space dynamics in Transformer residual streams. By projecting activations through logit-lens and measuring the entropy of decoded next-token distributions, the method reveals two core token-prediction strategies: expansion and pruning. Across model families, tasks, and output formats, entropy profiles exhibit systematic, depth-rescaled patterns, enabling family-specific fingerprints and task-format discriminability while remaining robust to Rényi-entropy variants. Interventions disrupting expansion phases tend to degrade downstream performance, indicating expansion often plays a more critical role for accuracy. Overall, Entropy-Lens advances mechanistic interpretability by linking information-theoretic measures to layer-wise decision strategies and information processing across depth, with implications for model analysis and design.

Abstract

In large language models (LLMs), each block operates on the residual stream to map input token sequences to output token distributions. However, most of the interpretability literature focuses on internal latent representations, leaving token-space dynamics underexplored. The high dimensionality and categoricity of token distributions hinder their analysis, as standard statistical descriptors are not suitable. We show that the entropy of logit-lens predictions overcomes these issues. In doing so, it provides a per-layer scalar, permutation-invariant metric. We introduce Entropy-Lens to distill the token-space dynamics of the residual stream into a low-dimensional signal. We call this signal the entropy profile. We apply our method to a variety of model sizes and families, showing that (i) entropy profiles uncover token prediction dynamics driven by expansion and pruning strategies; (ii) these dynamics are family-specific and invariant under depth rescaling; (iii) they are characteristic of task type and output format; (iv) these strategies have unequal impact on downstream performance, with the expansion strategy usually being more critical. Ultimately, our findings further enhance our understanding of the residual stream, enabling a granular assessment of how information is processed across model depth.

Figures (10)

• Figure 1: Fraction of top-$p$ ($p=0.6$) next-token candidates shared between a layer and its successor across different models. The consistently high overlap confirms that candidate sets remain stable across layers (Claim C2).
• Figure 2: Overview of the Entropy-Lens pipeline. Left: intermediate residual-stream activations at each layer are projected into the output space via logit-lens, yielding layer-wise next-token distributions whose entropy is computed. Right: the resulting per-token entropy profiles are aggregated across generated tokens (e.g., by concatenation or averaging) to produce a representation that captures how the model expands and prunes its space of plausible continuations across depth and can optionally be fed to a classifier to assess profile distinctiveness.
• Figure 3: t-SNE projection of aggregated entropy profiles obtained from a blank prompt for different model families. Profiles cluster by family rather than by model size, indicating family-specific token prediction dynamics. Point size scales with number of parameters.
• Figure 4: Average entropy profiles over 32 generated tokens for different model families. When depth is normalized, models within the same family exhibit aligned entropy trajectories, revealing family-specific coarse-to-fine token prediction dynamics.
• Figure 5: PCA projection of aggregated entropy profiles extracted from the topic–format dataset. Profiles cluster by output format, indicating that different formats are associated with characteristic entropy and token prediction dynamics.