Residual Stream Analysis of Overfitting And Structural Disruptions

Abstract

Ensuring that large language models (LLMs) remain both helpful and harmless poses a significant challenge: fine-tuning on repetitive safety datasets, where unsafe prompts are paired with standard refusal templates, often leads to false refusals, in which benign queries are declined. We first quantify this effect, showing that safety data exhibits substantially lower token entropy and 2-gram diversity (0.048) compared to general instruction data. To uncover the root cause, we introduce FlowLens, a stable PCA-based tool for residual-stream geometry analysis, and reveal that higher proportions of safety examples concentrate variance along a few components, reducing representational smoothness and driving false refusals (false refusal rate rises from 63 percent to 84 percent as safety data increases from 0 percent to 40 percent). Guided by these insights, we propose Variance Concentration Loss (VCL), an auxiliary regularizer that penalizes excessive variance concentration in mid-layer residuals. Empirical results demonstrate that VCL reduces false refusals by over 35 percentage points while maintaining or improving performance on general benchmarks such as MMLU and GSM8K.

Figures (9)

• Figure 1: Examples of false refusal on an exaggerated safety prompt sampled from XSTest. Our method avoids false refusal and gives an appropriate response. Model and dataset details are provided in Section \ref{['sec:setup']}.
• Figure 2: Residual trajectories of the mean alignment score along the top principal component using FlowLens for four instruction-tuned LLMs on TruthfulQA (General, blue) versus XSTest (Safe, red). Panels (a)–(d) correspond to: (a) Llama-3.2-1B, (b) Llama-3.1-8B-Instruct, (c) Llama-3.2-1B-SFT, and (d) Llama-3.2-1B-VCL (ours). Each curve plots the projection of the final token’s residual vector at normalized layer depth $[0,1]$. Examples shown above illustrates that standard safety fine-tuning collapses mid-layer variance around depths 0.4–0.6, leading to more false refusals on XSTest; in contrast, VCL stabilizes variance across layers and maintains safety.
• Figure 3: Loss behavior differences between safety and general tasks. Safety data shows lower average PPL but greater variance and heavier tail. Our experiments employ the Llama-3.1-Tulu-3-8B model family.
• Figure 4: Projections of residual trajectories using FlowLens for three instruction-tuned language models on the TruthfulQA dataset. Each point represents the PCA-projected residual vector of the final token from one prompt, colored by its corresponding layer index (depth normalized to $[0,1]$).
• Figure 5: Layerwise $\text{PC}_{\mathit{ID}}$ center trajectories under FlowLens. Top: safety vs. general prompts within the same instruction-tuned model (LLaMA-3.2-1B-Instruct); Bottom: models trained on domain-specific subsets from the Tülu 3 dataset lambert2024tulu3. Safety data produces irregular $\text{PC}_{\mathit{ID}}$ curves, deviating from the smooth, aligned progression seen in general and other domains. These deviations signal a breakdown in residual stream structure caused by safety fine-tuning.