Compression for Better: A General and Stable Lossless Compression Framework

TL;DR

The paper introduces a universal LossLess Compression (LLC) framework that defines a lossless compression neighborhood through total differentiation and Hessian-based bounds, enabling stable post-training compression without fine-tuning. It reformulates quantization as a grouped knapsack problem and decompositions under low-rank constraints within the LLC error neighborhood to guarantee $ar{f}(w) \le f(w)$, often improving loss. Extensive experiments across ImageNet, NLP benchmarks, and other datasets show LLC achieves lossless or loss-reducing compression for both quantization and decomposition with efficient search (e.g., per-layer bit-width selection in under 10 minutes on a single GPU) and without retraining. The work provides a principled, gradient-driven approach to compression boundaries and demonstrates practical impact for faster, smaller models with preserved or enhanced performance.

Abstract

This work focus on how to stabilize and lossless model compression, aiming to reduce model complexity and enhance efficiency without sacrificing performance due to compression errors. A key challenge is effectively leveraging compression errors and defining the boundaries for lossless compression to minimize model loss. i.e., compression for better. Currently, there is no systematic approach to determining this error boundary or understanding its specific impact on model performance. We propose a general \textbf{L}oss\textbf{L}ess \textbf{C}ompression theoretical framework (\textbf{LLC}), which further delineates the compression neighborhood and higher-order analysis boundaries through the total differential, thereby specifying the error range within which a model can be compressed without loss. To verify the effectiveness of LLC, we apply various compression techniques, including quantization and decomposition. Specifically, for quantization, we reformulate the classic quantization search problem as a grouped knapsack problem within the lossless neighborhood, achieving lossless quantization while improving computational efficiency. For decomposition, LLC addresses the approximation problem under low-rank constraints, automatically determining the rank for each layer and producing lossless low-rank models. We conduct extensive experiments on multiple neural network architectures on different datasets. The results show that without fancy tricks, LLC can effectively achieve lossless model compression. Our code will be made publicly.

Figures (2)

• Figure 1: In the analysis of noise boundaries for weights and activations, for activations, when the noise level is below $1 \times 10^{-3}$, only the first-order term needs to be considered, as higher-order terms have negligible impact on optimization. When the noise is below $8 \times 10^{-2}$, the second-order Hessian term should be incorporated along with the first-order gradient term in the optimization objective. For weights, although theoretically a well-trained model should yield zero weight gradients, in practice, the weight gradients are seldom exactly zero and therefore still need to be taken into account.
• Figure 2: Performance curves and loss curves of LLC in quantization and decomposition methods. LLC can achieve better performance with lower loss and smaller models.

Theorems & Definitions (2)

• Theorem 3.1
• Lemma 3.2