Enhanced Super-Resolution Training via Mimicked Alignment for Real-World Scenes

TL;DR

A novel plug-and-play module designed to mitigate misalignment issues by aligning LR inputs with HR images during training by mimicking a novel LR sample that aligns with HR while preserving the degradation characteristics of the original LR samples.

Abstract

Image super-resolution methods have made significant strides with deep learning techniques and ample training data. However, they face challenges due to inherent misalignment between low-resolution (LR) and high-resolution (HR) pairs in real-world datasets. In this study, we propose a novel plug-and-play module designed to mitigate these misalignment issues by aligning LR inputs with HR images during training. Specifically, our approach involves mimicking a novel LR sample that aligns with HR while preserving the degradation characteristics of the original LR samples. This module seamlessly integrates with any SR model, enhancing robustness against misalignment. Importantly, it can be easily removed during inference, therefore without introducing any parameters on the conventional SR models. We comprehensively evaluate our method on synthetic and real-world datasets, demonstrating its effectiveness across a spectrum of SR models, including traditional CNNs and state-of-the-art Transformers. The source codes will be publicly made available at https://github.com/omarAlezaby/Mimicked_Ali .

Figures (8)

• Figure 1: Motivation: Despite significant effort, misalignment issues persist in real-world datasets cai2019towardzhang2019zoom, limiting the potential of SR models. To address this challenge, we propose a novel alignment method using mimicked-LR, which maintains the same geometry and color properties as the HR input while sharing the same degradation type as the LR. By training our network with this newly generated mimicked-LR, we fully leverage the potential of SR models. $\widehat{HR}$ is the downscaled HR.
• Figure 2: The importance of Color and Geometrical Alignment between the LR and HR images for the training on Realistic SR Datasets
• Figure 3: Overview. In the training stage, we employ a modification module for alignment purposes to create a Mimicked-LR input with improved consistency with the GT. Such module generates or mimics a new LR ($Mim_{LR}$) which (i) has the same degradations as the LR input image, (ii) is geometrically aligned with the HR, and (iii) is color consistent with the GT (same colors, brightness, etc). During inference, we remove the generation/alignment module and replace the Mimicked-LR with the normal LR input, allowing for a sharper output without color changes or distortion.
• Figure 4: Illustration of the proposed LR Mimicking Architecture.
• Figure 5: Visualization of our Mimicking alignment quality. We compare to dense alignment using Optical Flow. The error maps illustrate that our method produces outputs better aligned with the HR image.