Image Restoration via Multi-domain Learning

TL;DR

This paper tackles the challenge of restoring natural images degraded by diverse phenomena under practical constraints by introducing SWFormer, a Transformer-like backbone guided by Spatial-Wavelet-Fourier priors. It integrates multi-domain learning into both inter-block and intra-block designs, featuring a Lossless Multi-Input Multi-Output framework, a Spatial-Wavelet-Fourier Mixer with three branches, and a Multi-scale ConvFFN to fuse domain features across resolutions. The approach yields state-of-the-art results across ten restoration tasks (e.g., dehazing, deraining, motion/defocus deblurring, shadow removal, underwater enhancement, and low-light enhancement) while offering favorable trade-offs in parameters, FLOPs, and latency. The extensive experiments and ablations demonstrate the practical impact of multi-domain priors for efficient, high-quality image restoration with broad applicability in real-world scenarios.

Abstract

Due to adverse atmospheric and imaging conditions, natural images suffer from various degradation phenomena. Consequently, image restoration has emerged as a key solution and garnered substantial attention. Although recent Transformer architectures have demonstrated impressive success across various restoration tasks, their considerable model complexity poses significant challenges for both training and real-time deployment. Furthermore, instead of investigating the commonalities among different degradations, most existing restoration methods focus on modifying Transformer under limited restoration priors. In this work, we first review various degradation phenomena under multi-domain perspective, identifying common priors. Then, we introduce a novel restoration framework, which integrates multi-domain learning into Transformer. Specifically, in Token Mixer, we propose a Spatial-Wavelet-Fourier multi-domain structure that facilitates local-region-global multi-receptive field modeling to replace vanilla self-attention. Additionally, in Feed-Forward Network, we incorporate multi-scale learning to fuse multi-domain features at different resolutions. Comprehensive experimental results across ten restoration tasks, such as dehazing, desnowing, motion deblurring, defocus deblurring, rain streak/raindrop removal, cloud removal, shadow removal, underwater enhancement and low-light enhancement, demonstrate that our proposed model outperforms state-of-the-art methods and achieves a favorable trade-off among restoration performance, parameter size, computational cost and inference latency. The code is available at: https://github.com/deng-ai-lab/SWFormer.

Figures (13)

• Figure 1: (a) Spatial-domain representations for various restoration tasks. (b) Performance and model complexity balance: Average PSNR vs. Params/FLOPs/Latency/MACs. We average performance across different benchmarks for statistical characteristics. (c) Motivation: Spatial-Wavelet-Fourier analysis for various degradations. For simplicity, we present the results in grayscale.
• Figure 2: Inter-block designs: (a) Single-Input Single-Output, (b) Multi-Input Multi-Output, (c) Lossless Multi-Input Multi-Output.
• Figure 3: Comparison of Token Mixer and FFN modules across various restoration methods at Intra-block level.
• Figure 4: The overall framework of our proposed SWFormer and its detailed components.
• Figure 5: The quantitative evaluation results on synthetic/real-world rain streak removal and raindrop removal.