Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compression

Minjun Kim, Jaehyeon Choi, Hyunwoo Yang, Jongjin Kim, Jinho Song, U Kang

Abstract

What happens when multiple compression methods are combined-does the order in which they are applied matter? Joint model compression has emerged as a powerful strategy to achieve higher efficiency by combining multiple methods such as pruning and quantization. A central but underexplored factor in joint model compression is the compression order, or the sequence of different methods within the compression pipeline. Most prior studies have either sidestepped the issue by assuming orthogonality between techniques, while a few have examined them only in highly constrained cases. Consequently, the broader role of compression order in shaping model performance remains poorly understood. In this paper, we address the overlooked problem of compression order and provide both theoretical and empirical analysis. We formulate the problem of optimizing the compression order and introduce the Progressive Intensity Hypothesis, which states that weaker perturbations should precede stronger ones. We provide theoretical guarantees showing that the relative benefit of one order increases with the underlying performance gap. Extensive experiments on both language and vision models validate the hypothesis, and further show its generality to broader setups such as multi-stage compression and mixed-precision quantization.

Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compression

Abstract

What happens when multiple compression methods are combined-does the order in which they are applied matter? Joint model compression has emerged as a powerful strategy to achieve higher efficiency by combining multiple methods such as pruning and quantization. A central but underexplored factor in joint model compression is the compression order, or the sequence of different methods within the compression pipeline. Most prior studies have either sidestepped the issue by assuming orthogonality between techniques, while a few have examined them only in highly constrained cases. Consequently, the broader role of compression order in shaping model performance remains poorly understood. In this paper, we address the overlooked problem of compression order and provide both theoretical and empirical analysis. We formulate the problem of optimizing the compression order and introduce the Progressive Intensity Hypothesis, which states that weaker perturbations should precede stronger ones. We provide theoretical guarantees showing that the relative benefit of one order increases with the underlying performance gap. Extensive experiments on both language and vision models validate the hypothesis, and further show its generality to broader setups such as multi-stage compression and mixed-precision quantization.
Paper Structure (27 sections, 2 theorems, 10 equations, 16 figures, 4 tables)

This paper contains 27 sections, 2 theorems, 10 equations, 16 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries and Related Works
  3. Joint Compression Order Optimization
  4. Problem Definition
  5. Characterizing Compression Attributes
  6. The Progressive Intensity Hypothesis
  7. Theoretical Analysis
  8. Experimental Findings
  9. Experimental Setup
  10. Analysis on Language Models
  11. Analysis on Vision Models
  12. Beyond Pruning and Quantization: Toward General Pipelines
  13. Conclusion
  14. Notation
  15. Details on Theoretical Analysis
  16. ...and 12 more sections

Key Result

Theorem 1

Suppose we compress a model $\phi$ with two compression methods $f_{1}(\cdot)$ and $f_{2}(\cdot)$ with respective granularities $t_{f_{1}}$ and $t_{f_{2}}$, where disjoint selectivity holds. Then, under Assumption assumption:general, the compression order advantage $\mathcal{A}(f_{1} \rightarrow f_{

Figures (16)

  • Figure 1: The Progressive Intensity Hypothesis: Given two compression techniques, we conjecture that compressed models perform better if the stronger method is applied after the weaker one. That said, the optimal order between pruning and quantization varies with their compression ratios.
  • Figure 2: A case study of pruning $\mathcal{P}(\cdot)$ and quantization $\mathcal{Q}(\cdot)$ on model $\phi$. (a) if pruning granularity (green) is coarser or equal to quantization granularity (orange), disjoint selectivity holds. (b) Otherwise, partial removal of quantization units by pruning introduces extra error, termed interference$\Delta$.
  • Figure 3: Across diverse language models, the compression order advantage $\mathcal{A}(\mathcal{Q} \rightarrow \mathcal{P})$ increases monotonically with the CER difference $C^{*}_{\mathcal{P}} - C_{\mathcal{Q}}$. See Section \ref{['subsec:experiment_llms']} for details.
  • Figure 4: Compression order advantage $\mathcal{A}(\mathcal{Q} \rightarrow \mathcal{P})$ against CER difference $C^{*}_{\mathcal{P}} - C_{\mathcal{Q}}$ for three pruning $\mathcal{P}(\cdot)$ and four quantization $\mathcal{Q}(\cdot)$ methods on a LLaMA 3 8B model. Our hypothesis consistently holds for language models regardless of pruning granularity, rotation, and weight updates.
  • Figure 5: Rotation impact on pruning. See Section \ref{['subsec:experiment_llms']} for details.
  • ...and 11 more figures

Theorems & Definitions (12)

  • Definition 1: Compression Granularity
  • Definition 2: Performance Gap
  • Definition 3: Compression Equivalent Ratio
  • Definition 4: Compression Order Advantage
  • Definition 5: Disjoint Selectivity
  • Theorem 1: Compression Order Advantage under Disjoint Selectivity
  • proof
  • Theorem 2: Monotonicity
  • proof
  • Definition 6: Interference
  • ...and 2 more