Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortion

Jianxiang Zang, Meiling Ning, Yongda Wei, Shihan Dou, Jiazheng Zhang, Nijia Mo, Binhong Li, Tao Gui, Qi Zhang, Xuanjing Huang

TL;DR

The paper addresses the mismatch between compression-based informatics and actual LM capabilities across architectures by introducing Compression Hacking, a geometric phenomenon where noise-dominated directions inflate perceived compression. It proposes three refined, geometry-aware metrics—spectral entropy compression, semantic coefficient of variation, and a PCS-based manifold correction—integrated into a train-free self-evaluation pipeline. Across 18 open-source LMs and six benchmarks, the refined metrics achieve Spearman correlations with comprehensive capabilities above 0.9, markedly outperforming prior metrics and demonstrating the importance of incorporating representational geometry. The work offers a robust, architecture-agnostic framework for interpreting LM intelligence and has potential implications for model compression, pruning, and evaluation methodologies.

Abstract

Recently, the concept of ``compression as intelligence'' has provided a novel informatics metric perspective for language models (LMs), emphasizing that highly structured representations signify the intelligence level of LMs. However, from a geometric standpoint, the word representation space of highly compressed LMs tends to degenerate into a highly anisotropic state, which hinders the LM's ability to comprehend instructions and directly impacts its performance. We found this compression-anisotropy synchronicity is essentially the ``Compression Hacking'' in LM representations, where noise-dominated directions tend to create the illusion of high compression rates by sacrificing spatial uniformity. Based on this, we propose three refined compression metrics by incorporating geometric distortion analysis and integrate them into a self-evaluation pipeline. The refined metrics exhibit strong alignment with the LM's comprehensive capabilities, achieving Spearman correlation coefficients above 0.9, significantly outperforming both the original compression and other internal structure-based metrics. This confirms that compression hacking substantially enhances the informatics interpretation of LMs by incorporating geometric distortion of representations.

Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortion

TL;DR

The paper addresses the mismatch between compression-based informatics and actual LM capabilities across architectures by introducing Compression Hacking, a geometric phenomenon where noise-dominated directions inflate perceived compression. It proposes three refined, geometry-aware metrics—spectral entropy compression, semantic coefficient of variation, and a PCS-based manifold correction—integrated into a train-free self-evaluation pipeline. Across 18 open-source LMs and six benchmarks, the refined metrics achieve Spearman correlations with comprehensive capabilities above 0.9, markedly outperforming prior metrics and demonstrating the importance of incorporating representational geometry. The work offers a robust, architecture-agnostic framework for interpreting LM intelligence and has potential implications for model compression, pruning, and evaluation methodologies.

Abstract

Recently, the concept of ``compression as intelligence'' has provided a novel informatics metric perspective for language models (LMs), emphasizing that highly structured representations signify the intelligence level of LMs. However, from a geometric standpoint, the word representation space of highly compressed LMs tends to degenerate into a highly anisotropic state, which hinders the LM's ability to comprehend instructions and directly impacts its performance. We found this compression-anisotropy synchronicity is essentially the ``Compression Hacking'' in LM representations, where noise-dominated directions tend to create the illusion of high compression rates by sacrificing spatial uniformity. Based on this, we propose three refined compression metrics by incorporating geometric distortion analysis and integrate them into a self-evaluation pipeline. The refined metrics exhibit strong alignment with the LM's comprehensive capabilities, achieving Spearman correlation coefficients above 0.9, significantly outperforming both the original compression and other internal structure-based metrics. This confirms that compression hacking substantially enhances the informatics interpretation of LMs by incorporating geometric distortion of representations.

Paper Structure

This paper contains 21 sections, 2 theorems, 23 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Compression Hacking
  3. Preliminary: Differential Entropy based Compression Metric
  4. Anisotropy: The Geometric Property Correlated with Compression
  5. Systematic Analysis
  6. Methodology
  7. Refined Metrics
  8. Evaluation Pipeline
  9. Experiments
  10. Setup
  11. Main Results
  12. Comparison with Baseline Metrics
  13. Effect of Anisotropy Razors
  14. Conclusion
  15. Limitations
  16. ...and 6 more sections

Key Result

Lemma B.1

Let $\Sigma \in \mathbb{R}^{D\times D}$ be the population covariance matrix and $\Sigma_{\mathbf{Z}} = \frac{1}{|\mathcal{V}|}\mathbf{Z}^\top\mathbf{Z}$ the sample covariance. The Ledoit-Wolf estimator attains minimal MSE when the shrinkage intensity satisfies $\beta_{\text{LW}} \asymp \frac{1}{|\mathcal{V}|}$. Under general covariance structures (without spectral sparsity), this yields asymptoti

Figures (9)

  • Figure 1: Comparison of compression metrics across different models and their corresponding ground-truth comprehensive capabilities, categorized into intra-family and cross-family comparisons.
  • Figure 2: Visualization of distribution of in-context representations and the eigenvalues across different models.
  • Figure 3: Regression fitting curves of compression versus anisotropy for different models, along with Mann-Whitney U tests between them. Here, **** denotes statistical significance at the 0.01% level.
  • Figure 4: Scatter plots of ground truth values across different models for the four metrics, along with fitted regression equations and Spearman correlation coefficients.
  • Figure 5: The qqplot of the eigenvalue distribution before and after using different anisotropy razors, and the distribution of the partition function $\mathcal{Z}(\mathbf{c})$.
  • ...and 4 more figures

Theorems & Definitions (3)

  • Lemma B.1: Asymptotic Optimality of Ledoit-Wolf Shrinkage ledoit2004well
  • Theorem B.2: Statistical Stability of the Principal Component Smoothing Estimator
  • proof