Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised Raindrop Removal from a Single Image using Conditional Diffusion Models

Lhuqita Fazry, Valentino Vito

TL;DR

The paper tackles raindrop removal from a single image by introducing DropWiper, a two-stage framework that first generates pseudo raindrop masks and then reconstructs the occluded background with a conditional diffusion model. It leverages residual-based masking and a synthetic raindrop detector trained via a refraction model on Cityscapes to guide diffusion-based inpainting, evaluated on the Raindrop dataset and Cityscapes. Results show that residual masks provide more reliable guidance than the detector due to domain shift, while the DDPM-based background reconstruction yields plausible restorations when masks are accurate. This work demonstrates the viability of diffusion models for single-image raindrop removal and underscores the need for better mask quality and domain adaptation.

Abstract

Raindrop removal is a challenging task in image processing. Removing raindrops while relying solely on a single image further increases the difficulty of the task. Common approaches include the detection of raindrop regions in the image, followed by performing a background restoration process conditioned on those regions. While various methods can be applied for the detection step, the most common architecture used for background restoration is the Generative Adversarial Network (GAN). Recent advances in the use of diffusion models have led to state-of-the-art image inpainting techniques. In this paper, we introduce a novel technique for raindrop removal from a single image using diffusion-based image inpainting.

Unsupervised Raindrop Removal from a Single Image using Conditional Diffusion Models

TL;DR

The paper tackles raindrop removal from a single image by introducing DropWiper, a two-stage framework that first generates pseudo raindrop masks and then reconstructs the occluded background with a conditional diffusion model. It leverages residual-based masking and a synthetic raindrop detector trained via a refraction model on Cityscapes to guide diffusion-based inpainting, evaluated on the Raindrop dataset and Cityscapes. Results show that residual masks provide more reliable guidance than the detector due to domain shift, while the DDPM-based background reconstruction yields plausible restorations when masks are accurate. This work demonstrates the viability of diffusion models for single-image raindrop removal and underscores the need for better mask quality and domain adaptation.

Abstract

Raindrop removal is a challenging task in image processing. Removing raindrops while relying solely on a single image further increases the difficulty of the task. Common approaches include the detection of raindrop regions in the image, followed by performing a background restoration process conditioned on those regions. While various methods can be applied for the detection step, the most common architecture used for background restoration is the Generative Adversarial Network (GAN). Recent advances in the use of diffusion models have led to state-of-the-art image inpainting techniques. In this paper, we introduce a novel technique for raindrop removal from a single image using diffusion-based image inpainting.
Paper Structure (18 sections, 4 equations, 10 figures)

This paper contains 18 sections, 4 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Raindrop Removal
  4. Diffusion Models
  5. Methodology
  6. Pseudo Raindrop Mask Generation
  7. Direct Thresholding on the Residual
  8. Raindrop Detection
  9. Background Reconstruction
  10. Experiments
  11. Experimental Datasets
  12. Experimental Settings
  13. Results and Discussion
  14. Residual Masks
  15. Synthetic Raindrops
  16. ...and 3 more sections

Figures (10)

  • Figure 1: Images with raindrops along with their corresponding ground-truth clean images from the Raindrop dataset qian2018
  • Figure 2: The diffusion model. The solid arrows denote reverse processes, whereas the dashed arrows denote forward processes
  • Figure 3: The raindrop detector network consists of $2$ convolutional blocks, followed by $6$ residual blocks, and ends up with $3$ up-sampling blocks. It takes an image with raindrops as input and then returns the raindrop mask as output
  • Figure 4: The refraction model. This model is a raindrop modeling technique based on ray tracking. Based on this model, the background image under the raindrop is a refracted manifestation obtained according to some degree and distance
  • Figure 5: The background restoration process. Given an image with raindrops and a pseudo-mask, the model reconstructs regions identified by the mask
  • ...and 5 more figures