Thinking in Granularity: Dynamic Quantization for Image Super-Resolution by Intriguing Multi-Granularity Clues

TL;DR

This work tackles the challenge of achieving high-quality image super-resolution under aggressive quantization by introducing Granular-DQ, a patch-wise, layer-invariant dynamic quantization framework. It combines a granularity-bit controller (GBC) that builds coarse-to-fine patch representations to assign bit-widths and an entropy-to-bit (E2B) mechanism that fine-tunes high-bit patches using entropy statistics. The approach avoids disturbing inter-layer relationships by not adapting layer-wise bit-widths to content, instead exploiting multi-granularity cues and information density to optimize the quantization process. Empirical results across CNN and transformer SR models demonstrate superior SR accuracy with significantly reduced BitOPs and FAB, enabling more practical SR on mobile and embedded devices. Overall, Granular-DQ offers a robust, content-aware quantization strategy that preserves reconstruction quality while greatly improving efficiency.

Abstract

Dynamic quantization has attracted rising attention in image super-resolution (SR) as it expands the potential of heavy SR models onto mobile devices while preserving competitive performance. Existing methods explore layer-to-bit configuration upon varying local regions, adaptively allocating the bit to each layer and patch. Despite the benefits, they still fall short in the trade-off of SR accuracy and quantization efficiency. Apart from this, adapting the quantization level for each layer individually can disturb the original inter-layer relationships, thus diminishing the representation capability of quantized models. In this work, we propose Granular-DQ, which capitalizes on the intrinsic characteristics of images while dispensing with the previous consideration for layer sensitivity in quantization. Granular-DQ conducts a multi-granularity analysis of local patches with further exploration of their information densities, achieving a distinctive patch-wise and layer-invariant dynamic quantization paradigm. Specifically, Granular-DQ initiates by developing a granularity-bit controller (GBC) to apprehend the coarse-to-fine granular representations of different patches, matching their proportional contribution to the entire image to determine the proper bit-width allocation. On this premise, we investigate the relation between bit-width and information density, devising an entropy-to-bit (E2B) mechanism that enables further fine-grained dynamic bit adaption of high-bit patches. Extensive experiments validate the superiority and generalization ability of Granular-DQ over recent state-of-the-art methods on various SR models. Code and supplementary statement can be found at \url{https://github.com/MmmingS/Granular-DQ.git}.

Figures (8)

• Figure 1: Visual comparison of (a) previous dynamic quantization pipeline hong2022cadyq that adapt the bit allocation for layers and patches simultaneously and (b) our Granular-DQ pipeline conducts patch-wise and layer-invariant dynamic quantization, which contains two steps: 1) granularity-aware bit allocation and 2) fine-grained bit-width adaption based on the entropy statistics. Our method recovers a better SR image with a lower average bit.
• Figure 2: The schematic of the proposed Granular-DQ for SR networks. Granular-DQ is a patch-wise and layer-invariant quantization pipeline, which contains two key steps: 1) granularity-aware bit allocation by the granularity-bit controller (GBC) and 2) entropy-based fine-grained bit-width adaption on the patches allocated with high bits in GBC based on an entropy-to-bit (E2B) mechanism. During the inference phase, the input image is partitioned into serial patches mapped to the adapted bit code, which forces the SR network to be specifically quantized for each patch.
• Figure 3: The structure of granularity-bit controller (GBC). It constructs hierarchical coarse-to-fine granularity representations for each patch. Then, it measures the granularity level of the patch upon its desired contribution percentage to the entire image, and maps this to quantization bit codes, finally achieving a tailored bit allocation.
• Figure 4: The generalized distribution statistic of the entropy for all LR patches on DIV2K.
• Figure 5: Qualitative comparison ($\times$4) on Urban100 and Test2K based on IDN and HAT-S models. Granular-DQ reconstructs SR images with better details and quantitative results