Beyond Image Super-Resolution for Image Recognition with Task-Driven Perceptual Loss

TL;DR

This work tackles recognition on low-resolution imagery by coupling super-resolution with high-level task guidance. It introduces Task-Driven Perceptual (TDP) loss to steer SR toward task-relevant high-frequency details, and a Cross-Quality Patch Mix (CQMix) with an alternating training schedule to avoid shortcut learning and domain gaps. Across semantic segmentation, object detection, and image classification, SR4IR demonstrates substantial performance gains over conventional SR baselines and many task-agnostic SR methods, approaching the oracle achieved with HR inputs. The framework is shown to generalize across SR backbones and datasets, producing perceptually pleasing SR outputs while significantly boosting downstream task accuracy, with practical training considerations and supplementary analyses supporting its robustness and effectiveness.

Abstract

In real-world scenarios, image recognition tasks, such as semantic segmentation and object detection, often pose greater challenges due to the lack of information available within low-resolution (LR) content. Image super-resolution (SR) is one of the promising solutions for addressing the challenges. However, due to the ill-posed property of SR, it is challenging for typical SR methods to restore task-relevant high-frequency contents, which may dilute the advantage of utilizing the SR method. Therefore, in this paper, we propose Super-Resolution for Image Recognition (SR4IR) that effectively guides the generation of SR images beneficial to achieving satisfactory image recognition performance when processing LR images. The critical component of our SR4IR is the task-driven perceptual (TDP) loss that enables the SR network to acquire task-specific knowledge from a network tailored for a specific task. Moreover, we propose a cross-quality patch mix and an alternate training framework that significantly enhances the efficacy of the TDP loss by addressing potential problems when employing the TDP loss. Through extensive experiments, we demonstrate that our SR4IR achieves outstanding task performance by generating SR images useful for a specific image recognition task, including semantic segmentation, object detection, and image classification. The implementation code is available at https://github.com/JaehaKim97/SR4IR.

Figures (12)

• Figure 1: Visualizations and comparisons of the results on semantic segmentation task with PASCAL VOC dataset pascal-voc-2012. (a) The ground-truth image and the class label map. (b) The bilinear-upsampled image and the predicted result when trained on bilinear-upsampled images. (c) the SR result of the SwinIR sr_swinir and the predicted result when trained on the SR images. (d) the SR result of the SwinIR and the predicted result when trained by the proposed SR4IR framework. We use DeepLabV3 chen2017rethinking as a task network and the downsampling scale factor is x4.
• Figure 2: Overview of the proposed SR4IR framework. Our SR4IR framework consists of two training phases, where SR and task networks are alternately trained. During the first phase, SR4IR updates the SR network using the TDP loss, which is introduced in Section \ref{['subsec:percep']}, while the task network is temporarily frozen. In the second phase, SR4IR updates the task network using the proposed data augmentation strategy called CQMix, which is introduced in Section \ref{['subsec:mix']}, while the SR network is temporarily frozen.
• Figure 3: Concept of CQMix. The black and white region in the figure represents $\mathbf{0}$ and $\mathbf{1}$ region in the binary mask $\boldsymbol{M}$.
• Figure 4: Visualization of images and semantic segmentation results on PASCAL VOC dataset. We present the restored images and the corresponding predicted segmentation maps, respectively. For (b) and (c), we use the SwinIR model with an SR scale factor of x4.
• Figure 5: Visualization of object detection results on PASCAL VOC dataset. The red box with orange annotation means the predicted object bounding box with the corresponding prediction. For (b) and (c), we use the SwinIR model with an SR scale factor of x4.