Prompt to Restore, Restore to Prompt: Cyclic Prompting for Universal Adverse Weather Removal

TL;DR

The paper tackles universal adverse weather removal (UAWR) by introducing CyclicPrompt, a cyclic prompt learning framework that unifies weather degradation handling in a single model. It fuses a Composite Context Prompt (C2P) with an Erase-and-Paste Mechanism (EPM) inside a transformer-based restoration network, leveraging weather-specific knowledge, textual context, and weather-free priors across two iterative passes. Empirically, CyclicPrompt achieves state-of-the-art results on synthetic and real-world datasets, outperforming methods like TransWeather, WeatherDiff, and T3-DiffWeather by notable margins (e.g., average PSNR improvements of over 0.3 dB) and demonstrating strong generalization across rain, fog, and snow. The approach offers a practical, efficient pathway toward robust outdoor vision under diverse adverse weather conditions by exploiting cyclic prompts, residual priors, and cross-attentive guidance.

Abstract

Universal adverse weather removal (UAWR) seeks to address various weather degradations within a unified framework. Recent methods are inspired by prompt learning using pre-trained vision-language models (e.g., CLIP), leveraging degradation-aware prompts to facilitate weather-free image restoration, yielding significant improvements. In this work, we propose CyclicPrompt, an innovative cyclic prompt approach designed to enhance the effectiveness, adaptability, and generalizability of UAWR. CyclicPrompt Comprises two key components: 1) a composite context prompt that integrates weather-related information and context-aware representations into the network to guide restoration. This prompt differs from previous methods by marrying learnable input-conditional vectors with weather-specific knowledge, thereby improving adaptability across various degradations. 2) The erase-and-paste mechanism, after the initial guided restoration, substitutes weather-specific knowledge with constrained restoration priors, inducing high-quality weather-free concepts into the composite prompt to further fine-tune the restoration process. Therefore, we can form a cyclic "Prompt-Restore-Prompt" pipeline that adeptly harnesses weather-specific knowledge, textual contexts, and reliable textures. Extensive experiments on synthetic and real-world datasets validate the superior performance of CyclicPrompt. The code is available at: https://github.com/RongxinL/CyclicPrompt.

Figures (9)

• Figure 1: Our CyclicPrompt is capable of removing multiple adverse weather degradations in one model and producing more visually pleasant images than several state-of-the-art UAWR methods: TransWeatherTransWeather, WeatherDiffWeatherDiff, and T$^3$-DiffweatherT3-DiffWeather. The quantitative comparisons in PSNR and SSIM also indicate the preferable performance of our method obviously.
• Figure 2: Overview of the proposed CyclicPrompt built upon a transformer architecture following a cyclic “Prompt-Restore-Prompt” process for UAWR. We first construct a composite context prompt (C2P) containing weather-specific knowledge, input-conditional vector, and textual prompt to perceive weather-related information and context-aware representations. After the C2P-driven UAWR, the erase-and-paste mechanism (EPM) leverages constrained restoration priors to substitute the weather-related part in C2P with the weather-free prompt to facilitate the decoder flow further. The residual prior is extracted to provide texture clues and structural appearance for better detail recovery.
• Figure 3: Illustration of the residual prior extraction. Each row corresponds to a representative sample under different adverse weather conditions. From left to right: (1) observed LQ images; (2) the initial restoration $\tilde{\textbf{I}}_{HQ}$ by C2P; and (3) the RCP extracted from $\tilde{\textbf{I}}_{HQ}$, which preserves degradation-free structural features with enhanced edge information.
• Figure 4: The 4-level transformer architecture in CyclicPrompt and each level contains several transformer blocks. The prompt block is inserted before each level of the decoder, where each contains two branches with the branch selection controlled by the restoration process: the 1st iteration with initial C2P (top branch) and the 2nd with cyclic C2P (bottom branch). We use the cross-hierarchy information mining blocks (CHIMB) CVHSSR to extract residual features.
• Figure 5: Visual Comparisons with different UAWR methods, i.e. TransWeather TransWeather, GrideFormer wang2024gridformer, T$^3$-DiffWeather T3-DiffWeather, WeatherDiff WeatherDiff and our CyclicPrompt on the RainDrop Dataset-Raindrop, Outdoor-Rain HRGAN and Snow100K-L Dataset-Snow100K datasets.