AdaIR: Adaptive All-in-One Image Restoration via Frequency Mining and Modulation

TL;DR

An adaptive all-in-one image restoration network based on frequency mining and modulation that achieves adaptive reconstruction by accentuating the informative frequency subbands according to different input degradations.

Abstract

In the image acquisition process, various forms of degradation, including noise, haze, and rain, are frequently introduced. These degradations typically arise from the inherent limitations of cameras or unfavorable ambient conditions. To recover clean images from degraded versions, numerous specialized restoration methods have been developed, each targeting a specific type of degradation. Recently, all-in-one algorithms have garnered significant attention by addressing different types of degradations within a single model without requiring prior information of the input degradation type. However, these methods purely operate in the spatial domain and do not delve into the distinct frequency variations inherent to different degradation types. To address this gap, we propose an adaptive all-in-one image restoration network based on frequency mining and modulation. Our approach is motivated by the observation that different degradation types impact the image content on different frequency subbands, thereby requiring different treatments for each restoration task. Specifically, we first mine low- and high-frequency information from the input features, guided by the adaptively decoupled spectra of the degraded image. The extracted features are then modulated by a bidirectional operator to facilitate interactions between different frequency components. Finally, the modulated features are merged into the original input for a progressively guided restoration. With this approach, the model achieves adaptive reconstruction by accentuating the informative frequency subbands according to different input degradations. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance on different image restoration tasks, including denoising, dehazing, deraining, motion deblurring, and low-light image enhancement. Our code is available at https://github.com/c-yn/AdaIR.

Figures (15)

• Figure 1: Left, from top to bottom: degraded images, ground-truth images, and the Fourier spectra of residual images obtained by subtracting the degraded images from the ground-truth images. The images are obtained from LOL-v1 lol, SOTS RESIDE, Rain100L RESIDE, GoPro gopro, and BSD68 bsd68, respectively. Right, the sub-graph illustrates the mean values of Fourier spectra on the square of length shown on the x-axis, across five tasks. The spectra are all resized to $320\times 320$ for comparisons. As seen, different tasks pay different attention to different frequency subbands. For example, there are larger discrepancies in low frequency between degraded and target image pairs of the low-light image enhancement and dehazing datasets. In contrast, the frequency differences are generally evenly distributed for image denoising.
• Figure 2: The t-SNE results of intermediate features produced by the three-task all-in-one models. Our model is better at learning discriminative degradation contexts.
• Figure 3: (a) The overall pipeline of the proposed AdaIR framework. It is a Transformer-based encoder-decoder architecture, employing novel Adaptive Frequency Learning Blocks (AFLB). Each AFLB contains (b) Frequency Mining Module (FMiM) that extracts different frequency components from input features guided by the adaptively decoupled spectra of the degraded input image, and (c) Frequency Modulation Module (FMoM) that exchanges the complementary information between different frequency features. (d) Cross Attention (CA). (e) Mask Generation Block (MGB) that yields a learnable frequency boundary for spectra decomposition. (f) H-L unit delivers high-frequency attention maps to enrich Low-frequency features. (g) L-H unit enhances high-frequency features by complementing it with low-frequency features. FFT and IFFT denote the Fast Fourier Transform and its inverse operator, respectively.
• Figure 4: Image dehazing comparisons on SOTS RESIDE between all-in-one methods. Compared to other algorithms, our method is more effective in haze removal.
• Figure 5: Image deraining results on Rain100L rain100L between all-in-one methods. AdaIR yields high-fidelity rain-free images with structural fidelity and without streak artifacts.