Disentangling Direction and Magnitude in Transformer Representations: A Double Dissociation Through L2-Matched Perturbation Analysis

TL;DR

This work examines how transformer representations encode information in terms of direction and magnitude by introducing L2-matched perturbation analysis, ensuring angular and magnitude perturbations produce identical Euclidean displacements. Applied to Pythia-family models, it uncovers a cross-over dissociation: angular perturbations primarily disrupt language modeling loss, while magnitude perturbations mainly impair syntactic processing, with causal interventions revealing two distinct mechanistic pathways (attention for direction, LayerNorm for magnitude). The study refines the linear representation hypothesis by showing partial functional specialization of geometric properties and highlights architecture dependence, including reversals in RMSNorm-based models. These findings have practical implications for interpretability and model editing, suggesting targeted interventions on direction versus magnitude to preserve specific capabilities and revealing the geometry of hidden-state representations as a structured driver of computation.

Abstract

Transformer hidden states encode information as high-dimensional vectors, yet whether direction (orientation in representational space) and magnitude (vector norm) serve distinct functional roles remains unclear. Studying Pythia-family models, we discover a striking cross-over dissociation: angular perturbations cause up to 42.9 more damage to language modeling loss, while magnitude perturbations cause disproportionately more damage to syntactic processing (20.4% vs.1.6% accuracy drop on subject-verb agreement).This finding is enabled by L2-matched perturbation analysis, a methodology ensuring that an gular and magnitude perturbations achieve identical Euclidean displacements. Causal intervention reveals that angular damage flows substantially through the attention pathways (28.4% loss recovery via attention repair), while magnitude damage flows partly through the LayerNorm pathways(29.9% recovery via LayerNorm repair). These patterns replicate across scales within the Pythia architecture family. These findings provide evidence that direction and magnitude support partially distinct computational roles in LayerNorm based architectures. The direction preferentially affects attentional routing, while magnitude modulates processing intensity for fine-grained syntactic judgments. We find different patterns in RMSNorm-based architectures, suggesting that the dissociation depends on architectural choices. Our results refine the linear representation hypothesis and have implications for model editing and interpretability research

Figures (7)

• Figure 1: Cross-over dissociation. (A) Loss damage across $\delta$ values: angular perturbations (red) cause up to 42.9$\times$ more damage than magnitude perturbations (blue) at small displacements. (B) BLiMP accuracy drop: the pattern reverses, with magnitude perturbations causing 12.8$\times$ greater accuracy loss at $\delta=5$. Error bars show standard error across 5 seeds.
• Figure 2: Dissociation replicates and amplifies at scale. (A) Angular/magnitude loss damage ratio (log scale) across perturbation magnitudes. The effect is 2 to 4$\times$ stronger in Pythia-1.4B (light blue) than Pythia-410M (dark blue). (B) Causal attention repair pattern replicates across scales: angular recovery consistently exceeds magnitude recovery (28.4% vs 15.2% at 410M; 23.8% vs 2.0% at 1.4B).
• Figure 3: Attention mediates angular damage. (A) Attention entropy change cascades through layers 8-16, with angular perturbations (red) causing 9.3$\times$ greater entropy increase than magnitude perturbations (blue) by layer 15. Shaded region indicates perturbed layers. (B) Causal attention repair recovers 28.4% of angular-induced loss damage versus only 15.2% for magnitude ($p=0.004$), confirming attention substantially mediates angular effects.
• Figure 4: Double dissociation in mechanistic pathways. (A) Attention repair preferentially recovers angular damage (28.4%) over magnitude damage (15.2%), $p=0.004$. (B) LayerNorm repair shows the inverse pattern: magnitude recovery (29.9%) exceeds angular recovery (13.7%), $p=0.002$. This mechanistic double dissociation parallels the behavioral dissociation in Figure \ref{['fig:crossover']}.
• Figure 5: Magnitude encodes parse tree depth. Pearson correlation between vector magnitude $\|\mathbf{h}\|$ and dependency parse depth. Baseline shows significant positive correlation ($r=0.142$, $p<10^{-45}$). Angular perturbation preserves this structure ($-11\%$), while magnitude perturbation destroys it ($-75\%$), explaining why magnitude perturbations specifically impair syntactic computation that depends on structural position.