All-in-one Weather-degraded Image Restoration via Adaptive Degradation-aware Self-prompting Model

TL;DR

This work develops an adaptive degradation-aware self-prompting model (ADSM) for all-in-one weather-degraded image restoration that employs the contrastive language-image pre-training model (CLIP) to facilitate the training of the proposed latent prompt generators (LPGs).

Abstract

Existing approaches for all-in-one weather-degraded image restoration suffer from inefficiencies in leveraging degradation-aware priors, resulting in sub-optimal performance in adapting to different weather conditions. To this end, we develop an adaptive degradation-aware self-prompting model (ADSM) for all-in-one weather-degraded image restoration. Specifically, our model employs the contrastive language-image pre-training model (CLIP) to facilitate the training of our proposed latent prompt generators (LPGs), which represent three types of latent prompts to characterize the degradation type, degradation property and image caption. Moreover, we integrate the acquired degradation-aware prompts into the time embedding of diffusion model to improve degradation perception. Meanwhile, we employ the latent caption prompt to guide the reverse sampling process using the cross-attention mechanism, thereby guiding the accurate image reconstruction. Furthermore, to accelerate the reverse sampling procedure of diffusion model and address the limitations of frequency perception, we introduce a wavelet-oriented noise estimating network (WNE-Net). Extensive experiments conducted on eight publicly available datasets demonstrate the effectiveness of our proposed approach in both task-specific and all-in-one applications.

Figures (13)

• Figure 1: Intuitive comparisons of the existing all-in-one approaches and our proposed method between the supervised and un-supervised metrical scores of reconstructed images. Our model notably enhances the naturalness and image quality of generated images, surpassing the capabilities of existing methods in both aspects.
• Figure 2: Overview of our proposed adaptive degradation-aware self-prompting model (ADSM) for all-in-one weather-degraded image restoration. Initially, we train three latent prompt generators (LPGs) to create the corresponding prompts for degradation type, degradation property and image caption. We then integrate these latent prompts into the diffusion model to perform iterative prompt learning, facilitating the weather degradation elimination and image content reconstruction.
• Figure 3: Two examples of texts that share similar meanings and depict the same image in the latent space. Despite their similar meanings, their latent encoding features differ.
• Figure 4: Illustration of our proposed latent prompt generators (LPGs). We employ the unlocked and locked image encoders of CLIP to generate the degradation prompt and caption prompt only from the processing weather-degraded images.
• Figure 5: Architectural illustration of our proposed wavelet-oriented noise estimating network (WNE-Net). We intentionally leave out the complexities of time embedding, degradation-aware prompts and caption prompts in our noise estimating network for simplification. Our WNE-Net follows a u-shaped architectural design, where the input consists of the combined state of the noisy signal $x_t$ at time $t$ and the degraded image condition $x_c$.