Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reclaiming Residual Knowledge: A Novel Paradigm to Low-Bit Quantization

Róisín Luo, Alexandru Drimbarean, James McDermott, Colm O'Riordan

TL;DR

CoRa reframes low-bit ConvNet quantization as an architecture-search problem over low-rank adapters to reclaim quantization residual knowledge, drastically reducing the search space compared with weight-space optimization. It introduces differentiable neural combinatorial optimization with a high-order normalized Butterworth kernel to select adapter ranks under a budget, enabling gradient-based solutions with minimal parameters. Empirically, CoRa achieves $4$-bit and $3$-bit quantization on ResNet variants that are competitive with QAT/PTQ baselines while requiring less than $250$ iterations on a $1600$-image calibration set, establishing a new standard for optimization efficiency. This training-free residual-knowledge reclamation pathway promises practical deployment gains for ConvNets on resource-constrained devices.

Abstract

This paper explores a novel paradigm in low-bit (i.e. 4-bits or lower) quantization, differing from existing state-of-the-art methods, by framing optimal quantization as an architecture search problem within convolutional neural networks (ConvNets). Our framework, dubbed \textbf{CoRa} (Optimal Quantization Residual \textbf{Co}nvolutional Operator Low-\textbf{Ra}nk Adaptation), is motivated by two key aspects. Firstly, quantization residual knowledge, i.e. the lost information between floating-point weights and quantized weights, has long been neglected by the research community. Reclaiming the critical residual knowledge, with an infinitesimal extra parameter cost, can reverse performance degradation without training. Secondly, state-of-the-art quantization frameworks search for optimal quantized weights to address the performance degradation. Yet, the vast search spaces in weight optimization pose a challenge for the efficient optimization in large models. For example, state-of-the-art BRECQ necessitates $2 \times 10^4$ iterations to quantize models. Fundamentally differing from existing methods, \textbf{CoRa} searches for the optimal architectures of low-rank adapters, reclaiming critical quantization residual knowledge, within the search spaces smaller compared to the weight spaces, by many orders of magnitude. The low-rank adapters approximate the quantization residual weights, discarded in previous methods. We evaluate our approach over multiple pre-trained ConvNets on ImageNet. \textbf{CoRa} achieves comparable performance against both state-of-the-art quantization-aware training and post-training quantization baselines, in $4$-bit and $3$-bit quantization, by using less than $250$ iterations on a small calibration set with $1600$ images. Thus, \textbf{CoRa} establishes a new state-of-the-art in terms of the optimization efficiency in low-bit quantization.

Reclaiming Residual Knowledge: A Novel Paradigm to Low-Bit Quantization

TL;DR

CoRa reframes low-bit ConvNet quantization as an architecture-search problem over low-rank adapters to reclaim quantization residual knowledge, drastically reducing the search space compared with weight-space optimization. It introduces differentiable neural combinatorial optimization with a high-order normalized Butterworth kernel to select adapter ranks under a budget, enabling gradient-based solutions with minimal parameters. Empirically, CoRa achieves -bit and -bit quantization on ResNet variants that are competitive with QAT/PTQ baselines while requiring less than iterations on a -image calibration set, establishing a new standard for optimization efficiency. This training-free residual-knowledge reclamation pathway promises practical deployment gains for ConvNets on resource-constrained devices.

Abstract

This paper explores a novel paradigm in low-bit (i.e. 4-bits or lower) quantization, differing from existing state-of-the-art methods, by framing optimal quantization as an architecture search problem within convolutional neural networks (ConvNets). Our framework, dubbed \textbf{CoRa} (Optimal Quantization Residual \textbf{Co}nvolutional Operator Low-\textbf{Ra}nk Adaptation), is motivated by two key aspects. Firstly, quantization residual knowledge, i.e. the lost information between floating-point weights and quantized weights, has long been neglected by the research community. Reclaiming the critical residual knowledge, with an infinitesimal extra parameter cost, can reverse performance degradation without training. Secondly, state-of-the-art quantization frameworks search for optimal quantized weights to address the performance degradation. Yet, the vast search spaces in weight optimization pose a challenge for the efficient optimization in large models. For example, state-of-the-art BRECQ necessitates iterations to quantize models. Fundamentally differing from existing methods, \textbf{CoRa} searches for the optimal architectures of low-rank adapters, reclaiming critical quantization residual knowledge, within the search spaces smaller compared to the weight spaces, by many orders of magnitude. The low-rank adapters approximate the quantization residual weights, discarded in previous methods. We evaluate our approach over multiple pre-trained ConvNets on ImageNet. \textbf{CoRa} achieves comparable performance against both state-of-the-art quantization-aware training and post-training quantization baselines, in -bit and -bit quantization, by using less than iterations on a small calibration set with images. Thus, \textbf{CoRa} establishes a new state-of-the-art in terms of the optimization efficiency in low-bit quantization.
Paper Structure (22 sections, 5 theorems, 42 equations, 14 figures, 1 table)

This paper contains 22 sections, 5 theorems, 42 equations, 14 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Method
  4. Differentiable relaxation
  5. Parameterized differentiable thresholding
  6. Neural combinatorial optimization
  7. Tricks for stable optimization
  8. Experiments
  9. Ablation study
  10. Comparing with baselines
  11. Extensive evaluation
  12. Performance scalability
  13. Hyper-parameter sensitivity
  14. Related work
  15. Future work
  16. ...and 7 more sections

Key Result

Theorem 1

Suppose a singular value decomposition given by: $(\Delta [\![W]\!]_n)_{(1)} = US_rV^T$ ($r=\rank(S_r)$). Then the factorization holds true: where $A=(S_r^{\frac{1}{2}}V^T)_{[1, r \times 1 \times 1]}$ and $B=(US_r^{\frac{1}{2}})_{[1, n \times k_1 \times k_2]}$. The $B \circledast A$ is referred as $r$-rank residual operator. The proof is provided in Appendix app:residual_proofs.

Figures (14)

  • Figure 1: CoRa framework: Searching for the optimal adapters, reclaiming the quantization residual knowledge, instead for the optimal quantized weights. The low-rank convolutional adapter at the $l$-th layer $B_{r_l}^{(l)} \circledast A_{r_l}^{(l)}$ is determined by a discrete integer $r_l$.
  • Figure 2: Differentiable thresholding with a high-order normalized Butterworth kernel $\Phi(r_l)$. The $S^{(l)}$ is in the $14$-th layer of a pre-trained resnet18 on ImageNet. The cut-off rank $r_l$ is $103$.
  • Figure 3: Optimization iterations and solution. The experiments are with resnet18 pre-trained on ImageNet.
  • Figure 4: Ablation study on ImageNet with resnet18. CoRa-H: Heuristic ranks. CoRa-O: Optimal ranks.
  • Figure 5: Optimization efficiency on ImageNet. Results are in a logarithmic scale.
  • ...and 9 more figures

Theorems & Definitions (7)

  • Theorem 1: Residual Convolutional Representation
  • Definition 1: Unfolding operator
  • Lemma 2: Tensor mode-$n$ product factorization
  • Lemma 3: Convolution mode-$n$ representation
  • Theorem 4: Convolution factorization
  • Corollary 5: Convolutional singular value decomposition
  • proof