DA2Diff: Exploring Degradation-aware Adaptive Diffusion Priors for All-in-One Weather Restoration

TL;DR

This paper addresses the challenge of all-in-one weather restoration across diverse degradations by introducing DA2Diff, a diffusion-based framework that learns degradation-aware priors via CLIP-based, learnable prompts. It combines a two-stage pipeline: degradation-aware prompt learning to capture weather-specific cues, and a prompt-guided diffusion restoration that injects these cues through Weather-specific Prompt Guidance and a Dynamic Expert Selection Modulator to adaptively activate restoration experts. The key contributions are (1) learnable weather prompts aligned in CLIP space, (2) a weather-specific prompt guidance module that integrates prompts into the diffusion encoder layers, and (3) a dynamic routing mechanism that flexibly dispatches restoration experts per input, improving performance and efficiency. Empirical results on synthetic and real-world datasets demonstrate superior restoration quality compared to state-of-the-art methods, with strong generalization to unseen degradations and realistic scenarios, highlighting practical impact for outdoor vision tasks.

Abstract

Image restoration under adverse weather conditions is a critical task for many vision-based applications. Recent all-in-one frameworks that handle multiple weather degradations within a unified model have shown potential. However, the diversity of degradation patterns across different weather conditions, as well as the complex and varied nature of real-world degradations, pose significant challenges for multiple weather removal. To address these challenges, we propose an innovative diffusion paradigm with degradation-aware adaptive priors for all-in-one weather restoration, termed DA2Diff. It is a new exploration that applies CLIP to perceive degradation-aware properties for better multi-weather restoration. Specifically, we deploy a set of learnable prompts to capture degradation-aware representations by the prompt-image similarity constraints in the CLIP space. By aligning the snowy/hazy/rainy images with snow/haze/rain prompts, each prompt contributes to different weather degradation characteristics. The learned prompts are then integrated into the diffusion model via the designed weather specific prompt guidance module, making it possible to restore multiple weather types. To further improve the adaptiveness to complex weather degradations, we propose a dynamic expert selection modulator that employs a dynamic weather-aware router to flexibly dispatch varying numbers of restoration experts for each weather-distorted image, allowing the diffusion model to restore diverse degradations adaptively. Experimental results substantiate the favorable performance of DA2Diff over state-of-the-arts in quantitative and qualitative evaluation. Source code will be available after acceptance.

Figures (8)

• Figure 1: Visual results generated by our DA$^{2}$Diff. Our method is capable of adaptively generating high-fidelity restoration results for the real-world weather degradations.
• Figure 2: The overall architecture of DA$^{2}$Diff. It involves two stages: degradation-aware prompt learning and prompt guidance diffusion restoration. In the first stage, we freeze the parameters of the image encoder and text encoder in CLIP and learn the weather prompts through contrastive learning. In the second stage, the learned weather prompts $P_{w}$ provide degradation-aware adaptive priors for diffusion-based restoration by two core components: (a) WPG and (b) DESM. WPG selects the most similar prompt $P_{s}$ from $P_{w}$, which matches the $i$-th state images $I_{t}$. Then, the weather-specific prompt $P_{s}$ is integrated into each encoder layer of the residual estimation model by PA and DESM. PA embeds the prompt $P_{s}$ into the feature map $F_{e}$ to generate degradation-aware features $F_{pa}$. Based on $F_{pa}$, DESM dynamically dispatches relevant restoration experts for the feature map $F_{e}$. Note that $F_{e}$ represents the output features of each encoder layer in the residual estimation model.
• Figure 3: The architecture of prompt adapter (PA). PA aims to integrate the weather-specific prompt $P_{s}$ into the feature map $F_{e}$, providing degradation-aware guidance for diffusion model.
• Figure 4: Visual comparisons on the RainDrop qian2018attentive test set. The region within the red box is zoomed for better comparison.
• Figure 5: Visual comparisons on the Test1 li2019heavy (rain + haze) set. The region within the red box is zoomed for better comparison.