Extreme Model Compression with Structured Sparsity at Low Precision

TL;DR

The paper tackles extreme model compression by unifying structured N:M sparsity with low-precision quantization, a combination that previously caused large accuracy losses. It introduces SLOPE, a training framework that adds a directional regularizer $L_{reg}(\mathbf{W}, \widehat{\mathbf{W}}) = \frac{1}{n} \sum_i (1 - \cos(\mathbf{w}_i, \widehat{\mathbf{w}}_i))$ to promote angular alignment between full-precision and sparse quantized weights, and uses an upper-bound-based objective to guide weight updates via straight-through estimation. Empirically, SLOPE achieves substantial compression (e.g., ~$20\times$ model size reduction) while preserving or exceeding full-precision accuracy on ResNet-18, DeiT-small, and Mask R-CNN across classification, detection, and segmentation tasks, outperforming state-of-the-art sparse and quantization baselines across 2:4 sparsity and low-bit settings. The work also provides theoretical insights into the performance bounds of structured sparse quantization and demonstrates that angular alignment, not mere Euclidean distance, is key to maintaining representational fidelity under extreme compression, enabling practical deployment of highly compact models.

Abstract

Deep neural networks (DNNs) are used in many applications, but their large size and high computational cost make them hard to run on devices with limited resources. Two widely used techniques to address this challenge are weight quantization, which lowers the precision of all weights, and structured sparsity, which removes unimportant weights while retaining the important ones at full precision. Although both are effective individually, they are typically studied in isolation due to their compounded negative impact on model accuracy when combined. In this work, we introduce SLOPE Structured Sparsity at Low Precision), a unified framework, to effectively combine structured sparsity and low-bit quantization in a principled way. We show that naively combining sparsity and quantization severely harms performance due to the compounded impact of both techniques. To address this, we propose a training-time regularization strategy that minimizes the discrepancy between full-precision weights and their sparse, quantized counterparts by promoting angular alignment rather than direct matching. On ResNet-18, SLOPE achieves $\sim20\times$ model size reduction while retaining $\sim$99% of the original accuracy. It consistently outperforms state-of-the-art quantization and structured sparsity methods across classification, detection, and segmentation tasks on models such as ResNet-18, ViT-Small, and Mask R-CNN.

Figures (5)

• Figure 1: Weight matrix representations under different compression settings. (a) Dense, full-precision weights offer high accuracy but are computationally expensive. (b) Structured 2:4 sparsity (every 4 elements contain 2 non-zeros) in full precision reduces the number of multiplications. (c) Quantization (4-bit) compresses memory usage. (d) Structured 2:4 quantization enables much higher inference speedups and compression ratios (See the Appendix).
• Figure 2: Accuracy vs. compression ratio on ResNet-18 models.
• Figure 3: Results of sparse 2:4 quantization on DeiT-small with ImageNet.
• Figure 4: The lower and upper bound trend when minimizing $\theta$.
• Figure 5: ResNet-18 performance across compression settings with and without SLOPE.

Theorems & Definitions (4)

• Theorem 1: Structured 2:4 Sparse Quantization Bounds
• proof
• Proposition 1: Both bounds coalesce as $\theta\to0$
• proof