Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models

TL;DR

This work presents a novel patch-based image restoration algorithm based on denoising diffusion probabilistic models that enables size-agnostic image restoration by using a guidedDenoising process with smoothed noise estimates across overlapping patches during inference.

Abstract

Image restoration under adverse weather conditions has been of significant interest for various computer vision applications. Recent successful methods rely on the current progress in deep neural network architectural designs (e.g., with vision transformers). Motivated by the recent progress achieved with state-of-the-art conditional generative models, we present a novel patch-based image restoration algorithm based on denoising diffusion probabilistic models. Our patch-based diffusion modeling approach enables size-agnostic image restoration by using a guided denoising process with smoothed noise estimates across overlapping patches during inference. We empirically evaluate our model on benchmark datasets for image desnowing, combined deraining and dehazing, and raindrop removal. We demonstrate our approach to achieve state-of-the-art performances on both weather-specific and multi-weather image restoration, and experimentally show strong generalization to real-world test images.

Figures (8)

• Figure 1: An overview of the forward diffusion (dashed line) and reverse denoising (solid line) processes for a conditional diffusion model.
• Figure 2: (a) Illustration of the patch-based diffusive image restoration pipeline detailed in Algorithm \ref{['alg:inference']}. (b) Illustrating mean estimated noise guided sampling updates for overlapping pixels across patches. We demonstrate a simplified example where $r=p/2$, and there are only four overlapping patches sharing the grid cell marked with the white border and gratings. In this case, we would perform sampling updates for the pixels in this region based on the mean estimated noise over the four overlapping patches, at each denoising time step $t$.
• Figure 3: Quantitative comparisons in terms of PSNR and SSIM (higher is better) with state-of-the-art image desnowing and deraining methods. Above half of the tables show comparisons of our weather-specific SnowDiff$_{p}$, RainHazeDiff$_{p}$ and RainDropDiff$_{p}$ models individually evaluated for each task. Bottom half of the tables show evaluations of our unified multi-weather model WeatherDiff$_{p}$ on all three test sets with respect to All-in-One Li:2020CVPR and TransWeather Valanarasu:2022CVPR multi-weather restoration methods. Best and second best values are indicated with bold text and underlined text respectively.
• Figure 4: Qualitative reconstruction comparisons of our best model on SnowTest100K test samples with DesnowNet liu2018desnownet and DDMSNet zhang2021deep.
• Figure 5: Qualitative reconstruction comparisons of our best model on Outdoor-Rain test samples with HRGAN li2019heavy and MPRNet liu2018desnownet.