Beyond Variance: Knowledge-Aware LLM Compression via Fisher-Aligned Subspace Diagnostics

TL;DR

This work addresses the challenge of preserving factual knowledge in post-training activation compression, where traditional variance-based methods like $SVD$ can discard low-variance but knowledge-carrying dimensions. It introduces Fisher-Aligned Subspace Compression (FASC), which optimizes a gradient-aware second-order objective and aligns activation subspaces with the empirical Fisher matrix $F=\mathbb{E}[gg^\top]$, solving a generalized eigenproblem to select loss-sensitive directions. A diagnostic measure, the Dependence Violation Score $\rho$, identifies knowledge-critical layers, enabling selective, high-impact compression with scalable randomized sketching. Empirically, FASC yields 6–8 percentage-point gains on knowledge benchmarks at 50% rank reduction across diverse models, and in some cases matches the factual recall of a much larger model, highlighting both practical deployment benefits and insights into how factual knowledge is organized in transformer networks.

Abstract

Post-training activation compression is essential for deploying Large Language Models (LLMs) on resource-constrained hardware. However, standard methods like Singular Value Decomposition (SVD) are gradient-blind: they preserve high-variance dimensions regardless of their impact on factual knowledge preservation. We introduce Fisher-Aligned Subspace Compression (FASC), a knowledge-aware compression framework that selects subspaces by directly modeling activation-gradient coupling, minimizing a second-order surrogate of the loss function. FASC leverages the Fisher Information Matrix to identify dimensions critical for factual knowledge, which often reside in low-variance but high-gradient-sensitivity subspaces. We propose the Dependence Violation Score (\r{ho}) as a general-purpose diagnostic metric that quantifies activation-gradient coupling, revealing where factual knowledge is stored within transformer architectures. Extensive experiments on Mistral-7B and Llama-3-8B demonstrate that FASC preserves 6-8% more accuracy on knowledge-intensive benchmarks (MMLU, LAMA) compared to variance-based methods at 50% rank reduction, effectively enabling a 7B model to match the factual recall of a 13B uncompressed model. Our analysis reveals that \r{ho} serves as a fundamental signal of stored knowledge, with high-\r{ho} layers emerging only when models internalize factual associations during training.

Figures (5)

• Figure 1: Conceptual comparison: SVD preserves high-variance (syntactic) dimensions; FASC preserves low-variance but gradient-critical (factual) dimensions.
• Figure 2: Layer-wise sensitivity on LAMA for Mistral-7B at 50% rank. SVD causes severe loss in mid-to-late layers (15--25); FASC maintains robust performance.
• Figure 3: Heatmap of $\rho$ across 32 layers of Mistral-7B. Higher $\rho$ (darker) indicates stronger activation–gradient coupling, concentrated in middle-to-late layers.
• Figure 4: Correlation between $\rho$ and FASC performance gain across Mistral-7B layers. Strong positive correlation ($r = 0.73$, $p < 0.001$) confirms $\rho$ identifies critical layers. Dashed line: $\rho = 0.3$ threshold.
• Figure 5: Principal angle analysis between FASC and SVD subspaces on Mistral-7B. High-$\rho$ layers ($\rho > 0.3$) diverge substantially (median $45^\circ$); low-$\rho$ layers align (median $12^\circ$).