Accelerating Image Super-Resolution Networks with Pixel-Level Classification

TL;DR

This work tackles the efficiency gap in large-scale SR by introducing PCSR, which allocates computation at the pixel level using a backbone, a lightweight pixel-level classifier, and multiple upsamplers of varying capacity. By predicting per-pixel class probabilities $p_{i,j}$ and selecting upsampler costs via $\text{cost}(U_j)$, PCSR achieves favorable PSNR–FLOP trade-offs and supports multi-scale SR through LIIF-based upsampling. The framework includes adaptive decision making (ADM) via clustering and a pixel-wise refinement step to reduce cross-boundary artifacts, with a multi-stage training regimen that stabilizes learning. Empirically, PCSR outperforms patch-based methods on Test2K/Test4K/Test8K and Urban100 across several backbones, and ablations confirm the benefits of pixel-level allocation, modest class counts ($M=2$), and refinement, highlighting its practical potential for real-world, large-image SR tasks.

Abstract

In recent times, the need for effective super-resolution (SR) techniques has surged, especially for large-scale images ranging 2K to 8K resolutions. For DNN-based SISR, decomposing images into overlapping patches is typically necessary due to computational constraints. In such patch-decomposing scheme, one can allocate computational resources differently based on each patch's difficulty to further improve efficiency while maintaining SR performance. However, this approach has a limitation: computational resources is uniformly allocated within a patch, leading to lower efficiency when the patch contain pixels with varying levels of restoration difficulty. To address the issue, we propose the Pixel-level Classifier for Single Image Super-Resolution (PCSR), a novel method designed to distribute computational resources adaptively at the pixel level. A PCSR model comprises a backbone, a pixel-level classifier, and a set of pixel-level upsamplers with varying capacities. The pixel-level classifier assigns each pixel to an appropriate upsampler based on its restoration difficulty, thereby optimizing computational resource usage. Our method allows for performance and computational cost balance during inference without re-training. Our experiments demonstrate PCSR's advantage over existing patch-distributing methods in PSNR-FLOP trade-offs across different backbone models and benchmarks. The code is available at https://github.com/3587jjh/PCSR.

Figures (8)

• Figure 1: The SR result on the image "1228" (Test2K), $\times$4. By adaptively distributing computational resources in a pixel-wise manner, our method can reduce the overall computational costs in terms of FLOPs compared to the patch-distributing method, while also achieving a better PSNR score.
• Figure 1: Difficulty density curve for the image “0855” (DIV2K) with $M$=2 on $\times$4. The range of values are divided into 100 bins, with density calculated as the count of values per bin divided by the total value count. The density, associated with each bin's center, is interpolated to form a smooth curve. Each dotted line indicates threshold for assigning pixels: pixels left of a line go to the light upsampler, those to the right to the heavy upsampler. The black dotted line represents a threshold ($=0.5$) of simple approach (i.e., allocating pixels to the upsampler with the highest probability), while red dotted line indicates an adaptively determined threshold by ADM.
• Figure 2: Visual comparison of PSNR and FLOPs between ClassSR, ARM, and PCSR (ours) on Test2K at scale $\times$4.
• Figure 2: Qualitative results of the previous methods chen2022armkong2021classsr and our method with $\times$4 SR on Test2K.
• Figure 3: The architecture of the proposed PCSR model when the number of classes $M$ is 2. We denote $q$ as a single query pixel in the HR space and $x_q$ for its coordinate. Pixel-level probabilities obtained from the classifier are used to allocate each query pixel to a suitably-sized upsampler for the prediction of its RGB value.