RefQSR: Reference-based Quantization for Image Super-Resolution Networks

TL;DR

RefQSR tackles the computational burden of deep SISR by introducing reference-based quantization that exploits image self-similarity. It clusters image patches into high-bit reference patches and low-bit query patches, and employs RefER to refine query quantization using warped reference features via a 4D cost volume and soft-argmax. The approach is modular and compatible with existing SR quantizers, delivering substantial BitOPs reductions (often exceeding 30–70%) while preserving PSNR/SSIM across SRResNet, CARN, and ELAN on common benchmarks. This yields practical SR acceleration suitable for resource-constrained devices, with potential extensions to end-to-end learning and other low-level vision tasks.

Abstract

Single image super-resolution (SISR) aims to reconstruct a high-resolution image from its low-resolution observation. Recent deep learning-based SISR models show high performance at the expense of increased computational costs, limiting their use in resource-constrained environments. As a promising solution for computationally efficient network design, network quantization has been extensively studied. However, existing quantization methods developed for SISR have yet to effectively exploit image self-similarity, which is a new direction for exploration in this study. We introduce a novel method called reference-based quantization for image super-resolution (RefQSR) that applies high-bit quantization to several representative patches and uses them as references for low-bit quantization of the rest of the patches in an image. To this end, we design dedicated patch clustering and reference-based quantization modules and integrate them into existing SISR network quantization methods. The experimental results demonstrate the effectiveness of RefQSR on various SISR networks and quantization methods.

Figures (10)

• Figure 1: Comparisons between the baseline SR network quantization methods and those integrated with the proposed method (denoted with -RefQSR) on $\times$4 SR of the Urban100 dataset for the SRResNet baseline. The endpoints in dashed lines correspond to several high- and low-bit combinations of RefQSR. RefQSR achieves a better trade-off between computational complexity and SR performance, e.g., displaying 35.9% to 76.9% BitOp savings over the baselines at similar PSNRs. CADyQ assigns an image-dependent dynamic bit $\delta$.
• Figure 2: Examples of patch-wise bit selections of (b) CADyQ and (c) CADyQ-RefQSR for the SRResNet baseline, where the darker the patches, the higher the average bit precision. Note that CADyQ allocates higher bit precision to many patches with textual details, whereas CADyQ-RefQSR utilizes image self-similarity to prevent assigning high-bit precision to similar patches.
• Figure 3: Overview of the proposed RefQSR framework. RefQSR divides the input image into patches and clusters them as reference and query patches via ClustBlock. The ResBlocks are processed with high-bit and low-bit quantization for reference and query patches, respectively. The RefER blocks are used to restore the quantization error of the query patches using the features from the reference patch. Finally, the output reference and query patches are merged to reconstruct the final HR image.
• Figure 4: A toy example illustrating the patch clustering procedure, where the number of patches $N$ is 5, and the number of texture classes $\hat{C}$ is 3. The five patches are grouped into two clusters according to the argmax indices, and the two patches that have the argmax indices at their positions are identified as reference patches.
• Figure 5: Several images (left) and their clustering results (right): Patches with the same color belong to the same cluster. The reference patches are marked with R, where the first three samples include isolated reference patches.