Uncertainty-Aware Source-Free Adaptive Image Super-Resolution with Wavelet Augmentation Transformer

TL;DR

A novel wavelet-based augmentation method, named Wavelet Augmentation Transformer (WAT), which can be flexibly incorporated with existing networks, to implicitly produce useful augmented data, and several regularization losses are proposed to constrain target LR and SR images in the frequency domain.

Abstract

Unsupervised Domain Adaptation (UDA) can effectively address domain gap issues in real-world image Super-Resolution (SR) by accessing both the source and target data. Considering privacy policies or transmission restrictions of source data in practical scenarios, we propose a SOurce-free Domain Adaptation framework for image SR (SODA-SR) to address this issue, i.e., adapt a source-trained model to a target domain with only unlabeled target data. SODA-SR leverages the source-trained model to generate refined pseudo-labels for teacher-student learning. To better utilize pseudo-labels, we propose a novel wavelet-based augmentation method, named Wavelet Augmentation Transformer (WAT), which can be flexibly incorporated with existing networks, to implicitly produce useful augmented data. WAT learns low-frequency information of varying levels across diverse samples, which is aggregated efficiently via deformable attention. Furthermore, an uncertainty-aware self-training mechanism is proposed to improve the accuracy of pseudo-labels, with inaccurate predictions being rectified by uncertainty estimation. To acquire better SR results and avoid overfitting pseudo-labels, several regularization losses are proposed to constrain target LR and SR images in the frequency domain. Experiments show that without accessing source data, SODA-SR outperforms state-of-the-art UDA methods in both synthetic$\rightarrow$real and real$\rightarrow$real adaptation settings, and is not constrained by specific network architectures.

Figures (7)

• Figure 1: PSNR vs. the usage of source data on the DRealSR wei2020component dataset. The less source data a method uses, the more restrictions it faces. SFDA and SSL represent source-free domain adaption and self-supervised learning methods respectively.
• Figure 2: Illustration of 2-level haar wavelet packet transform (WPT). WPT employs low-pass filters $H_{low}$ and high-pass filters $H_{high}$ in a recursive manner to decompose the original features into multiple sub-bands at different frequency resolutions.
• Figure 3: Architecture of the proposed SODA-SR framework. One target LR input image together with its seven geometrically augmented images (i.e., rotate and flip the input) will be fed into the teacher model to generate the refined pseudo-label. The Softmax normalization function in the teacher model will be replaced by Gumbel-Softmax gumbel. For one LR input image, the teacher model will run multiple times to generate $N$ pseudo-labels and calculate their mean and variance for uncertainty estimation.
• Figure 4: Wavelet Augmentation Transformer (WAT).
• Figure 5: Visual comparison for $\times$ 4 SR on DRealSR dataset (Sony$\rightarrow$Panasonic). Best viewed with zoom in.