A Comparative Study of Image Restoration Networks for General Backbone Network Design

TL;DR

The paper addresses the problem of limited task generality in image restoration backbones by benchmarking five representative networks across five degradation-driven tasks and analyzing how task-specific requirements shape backbone effectiveness. It formalizes a general-backbone design principle and proposes X-Restormer, a simple yet effective enhancement of Restormer that injects spatial self-attention into a U-shaped framework without increasing parameter count, thereby improving spatial mapping and cross-task performance. The authors demonstrate through extensive experiments that X-Restormer achieves state-of-the-art results on SR, denoising, deblurring, deraining, and dehazing, and exhibits superior all-in-one restoration generality. These findings suggest that a carefully balanced combination of architectural features and attention mechanisms can deliver robust, multi-task restoration without bespoke tuning for each task, potentially reducing development costs and enabling broader practical impact.

Abstract

Despite the significant progress made by deep models in various image restoration tasks, existing image restoration networks still face challenges in terms of task generality. An intuitive manifestation is that networks which excel in certain tasks often fail to deliver satisfactory results in others. To illustrate this point, we select five representative networks and conduct a comparative study on five classic image restoration tasks. First, we provide a detailed explanation of the characteristics of different image restoration tasks and backbone networks. Following this, we present the benchmark results and analyze the reasons behind the performance disparity of different models across various tasks. Drawing from this comparative study, we propose that a general image restoration backbone network needs to meet the functional requirements of diverse tasks. Based on this principle, we design a new general image restoration backbone network, X-Restormer. Extensive experiments demonstrate that X-Restormer possesses good task generality and achieves state-of-the-art performance across a variety of tasks.

Figures (5)

• Figure 1: Relative performance difference of different backbone networks on five image restoration tasks. The existing representative networks exhibit diverse performance on these tasks, while our method presents superior task generality.
• Figure 2: Selected five representative image restoration tasks with various degradation.
• Figure 3: The core operators in image restoration networks.
• Figure 4: Visual and LAM lam comparisons between Restormer and SwinIR. The LAM results and DI values indicate that Restormer exploits significantly more information than SwinIR. However, SwinIR reconstructs much more details than Restormer.
• Figure 5: The network structure of X-Restormer. To enhance the spatial mapping ability of Restormer and create a more general network, we replace half of the transposed self-attention blocks in Restormer with spatial self-attention blocks. For TSA, we retain the preliminary multi-Dconv head transposed attention (MDTA) used in Restormer. For SSA, we adopt the overlapping cross-attention (OCA) in HAT hat.