AdaBM: On-the-Fly Adaptive Bit Mapping for Image Super-Resolution

TL;DR

Image super-resolution models achieve high fidelity at the cost of heavy computation. AdaBM introduces an on-the-fly adaptive bit-mapping framework with two decoupled policies: image-wise bit adaptation and layer-wise adaptation, calibrated with a small LR calibration set and without full QAT. The method includes bit-aware clipping and a targeted finetuning routine using reconstruction losses, achieving competitive accuracy while reducing processing time by up to about 2000x compared to prior adaptive methods. This enables practical deployment of adaptive SR quantization across devices, balancing accuracy and efficiency for diverse inputs.

Abstract

Although image super-resolution (SR) problem has experienced unprecedented restoration accuracy with deep neural networks, it has yet limited versatile applications due to the substantial computational costs. Since different input images for SR face different restoration difficulties, adapting computational costs based on the input image, referred to as adaptive inference, has emerged as a promising solution to compress SR networks. Specifically, adapting the quantization bit-widths has successfully reduced the inference and memory cost without sacrificing the accuracy. However, despite the benefits of the resultant adaptive network, existing works rely on time-intensive quantization-aware training with full access to the original training pairs to learn the appropriate bit allocation policies, which limits its ubiquitous usage. To this end, we introduce the first on-the-fly adaptive quantization framework that accelerates the processing time from hours to seconds. We formulate the bit allocation problem with only two bit mapping modules: one to map the input image to the image-wise bit adaptation factor and one to obtain the layer-wise adaptation factors. These bit mappings are calibrated and fine-tuned using only a small number of calibration images. We achieve competitive performance with the previous adaptive quantization methods, while the processing time is accelerated by x2000. Codes are available at https://github.com/Cheeun/AdaBM.

Figures (4)

• Figure 1: Illustration of our adaptive bit-mapping. During inference, an input image is mapped to the image-wise bit adaptation factor based on its complexity (Image2Bit Mapping), then together with the layer-wise bit adaptation factors pre-determined based on layer-wise sensitivities (Layer2Bit Mapping), the two factors adapt the quantization bit-widths. Higher bit-widths are assigned to sensitive layers and complex images. The thresholds of mapping modules are calibrated and fine-tuned using a small set of calibration images.
• Figure 2: Analysis on layer-wise and image-wise quantization sensitivity. (a) Mean squared error (MSE) between statically quantized activation and corresponding FP is different per layer and per input image. (b) The relative orders of layer-wise MSEs tend to be invariant of the input image, as layer-wise MSEs of different images have high cosine similarity values. (c) Similarly, the relative orders of image-wise MSEs are consistent throughout network layers. This indicates that layer-wise bit adaptation and image-wise bit adaptation can be done separately, which effectively reduces the search cost for bit allocation.
• Figure 3: Qualitative results on Urban100 with 4-bit EDSR-based models. More results are provided in the supplementary document.
• Figure 4: Visualization of adaptive bit-mapping of AdaBM. Results are from EDSR-AdaBM (w8 a6MP).