AllRestorer: All-in-One Transformer for Image Restoration under Composite Degradations

TL;DR

An All-in-One Transformer Block (AiOTB), which adaptively removes all degradations present in a given image by modeling the relationships between all degradations and the image embedding in latent space, is introduced.

Abstract

Image restoration models often face the simultaneous interaction of multiple degradations in real-world scenarios. Existing approaches typically handle single or composite degradations based on scene descriptors derived from text or image embeddings. However, due to the varying proportions of different degradations within an image, these scene descriptors may not accurately differentiate between degradations, leading to suboptimal restoration in practical applications. To address this issue, we propose a novel Transformer-based restoration framework, AllRestorer. In AllRestorer, we enable the model to adaptively consider all image impairments, thereby avoiding errors from scene descriptor misdirection. Specifically, we introduce an All-in-One Transformer Block (AiOTB), which adaptively removes all degradations present in a given image by modeling the relationships between all degradations and the image embedding in latent space. To accurately address different variations potentially present within the same type of degradation and minimize ambiguity, AiOTB utilizes a composite scene descriptor consisting of both image and text embeddings to define the degradation. Furthermore, AiOTB includes an adaptive weight for each degradation, allowing for precise control of the restoration intensity. By leveraging AiOTB, AllRestorer avoids misdirection caused by inaccurate scene descriptors, achieving a 5.00 dB increase in PSNR compared to the baseline on the CDD-11 dataset.

Figures (11)

• Figure 1: AllRestorer can be applied to several image restoration tasks. (i) All-in-One Image Restoration: AllRestorer can address multiple single degradation scenarios with just one set of parameters. (ii) Real-World Image Restoration: AllRestorer can successfully respond to real-world all-in-one restoration and composite degradation restoration challenges. (iii) Composite Degradation Restoration: AllRestorer can be applied to various composite degradation restoration tasks with only a single network.
• Figure 2: AllRestorer achieves state-of-the-art performance on the CDD-11 dataset while remaining lightweight.
• Figure 3: Scene descriptor comparison between general all-in-one restoration methods and ours.
• Figure 4: Architecture of AllRestorer. (A) AllRestorer obtains adaptive weights and composite scene descriptors using the fixed CLIP encoder modeling the relationship between each descriptor in the memory bank and the impaired image. (B) In the overall pipeline of AllRestorer, AiOA in AiOTB introduces restoration solutions based on the proportion of each degradation via all composite scene descriptors and adaptive weights. SA in AiOTB adaptively removes all degradations according to the introduced restoration scheme.
• Figure 5: Adaptive weights for each degradation.