MetaWeather: Few-Shot Weather-Degraded Image Restoration

TL;DR

MetaWeather addresses the generalization gap in weather-degraded image restoration by proposing a universal few-shot adaptive framework that treats unseen weather as a task to be adapted from a small support set. It introduces spatial-channel matching within a patch-based, matching-based meta-learning design to extract degradation patterns and apply them to restore the query image via a single encoder–decoder with hierarchical structure, modeling weather effects as a degradation pattern $\\mathcal{G}(\\mathbf{X})$ that yields $\\mathbf{Y}=\\mathbf{X}-\\mathcal{G}(\\mathbf{X})$. The approach uses a Swin Transformer encoder, a MDTA-based matching module, and a two-branch decomposition of weather effects to achieve strong results on BID Task II.A, SPA-Data, and RealSnow, outperforming state-of-the-art multi-weather restoration methods. The findings demonstrate data-efficient, weather-agnostic restoration with flexible adaptation to unseen conditions, implying practical applicability for real-world outdoor vision tasks, supported by an explicit degradation model $\\mathbf{X}=\\mathbf{T} \\odot (\\mathbf{Y} + \\mathbf{P}) + (1 - \\mathbf{T}) \\odot \\mathbf{A}$ and a predictive mechanism for $\\mathcal{G}(\\mathbf{X})$ across diverse weather patterns.

Abstract

Real-world weather conditions are intricate and often occur concurrently. However, most existing restoration approaches are limited in their applicability to specific weather conditions in training data and struggle to generalize to unseen weather types, including real-world weather conditions. To address this issue, we introduce MetaWeather, a universal approach that can handle diverse and novel weather conditions with a single unified model. Extending a powerful meta-learning framework, MetaWeather formulates the task of weather-degraded image restoration as a few-shot adaptation problem that predicts the degradation pattern of a query image, and learns to adapt to unseen weather conditions through a novel spatial-channel matching algorithm. Experimental results on the BID Task II.A, SPA-Data, and RealSnow datasets demonstrate that the proposed method can adapt to unseen weather conditions, significantly outperforming the state-of-the-art multi-weather image restoration methods.

Figures (10)

• Figure 1: (a). The state-of-the-art image restoration models perform well on the seen weather types in the training data, including rain zamir2021multi, snow Liu2018desnow, raindrop qian2018attentive, and fog sakaridis2018semantic. (b). In contrast, these models show significant performance degradation on unseen weather types outside the scope of the training data, such as real rain (SPA-Data Wang2019spa) and real snow (RealSnow zhu2023learning), as well as co-occurrence of the weather types, rain, fog, and raindrops, in the training data (BID II.A Case 5 han2022blind). Best viewed with zoom and color.
• Figure 2: Overall architecture of MetaWeather. MetaWeather consists of a hierarchical encoder-decoder design and matching module. Our matching module matches the degradation pattern between query and support set images, which enables MetaWeather to fully utilize a few-shot support set. The matching results are passed on to the decoder blocks at each level, and the extracted degradation pattern of the query image is then subtracted from the query image, resulting in the clean query image.
• Figure 3: Attention modules in our matching module.
• Figure 4: Qualitative comparison on BID Task II.A dataset han2022blind. The results of top five baselines in \ref{['tab:main_table']} are sampled. Best viewed with zoom and color.
• Figure 5: Qualitative comparison on SPA-Data Wang2019spa and RealSnow zhu2023learning dataset. The results of top three baselines in \ref{['tab:real_world']} are sampled. Best viewed with zoom and color.