End-to-end Inception-Unet based Generative Adversarial Networks for Snow and Rain Removals

TL;DR

A global framework that consists of two Generative Adversarial Networks (GANs) is proposed where each handles the removal of each particle individually where each achieves significant improvements in comparison to the state-of-the-art approaches when tested on both synthetic and realistic datasets.

Abstract

The superior performance introduced by deep learning approaches in removing atmospheric particles such as snow and rain from a single image; favors their usage over classical ones. However, deep learning-based approaches still suffer from challenges related to the particle appearance characteristics such as size, type, and transparency. Furthermore, due to the unique characteristics of rain and snow particles, single network based deep learning approaches struggle in handling both degradation scenarios simultaneously. In this paper, a global framework that consists of two Generative Adversarial Networks (GANs) is proposed where each handles the removal of each particle individually. The architectures of both desnowing and deraining GANs introduce the integration of a feature extraction phase with the classical U-net generator network which in turn enhances the removal performance in the presence of severe variations in size and appearance. Furthermore, a realistic dataset that contains pairs of snowy images next to their groundtruth images estimated using a low-rank approximation approach; is presented. The experiments show that the proposed desnowing and deraining approaches achieve significant improvements in comparison to the state-of-the-art approaches when tested on both synthetic and realistic datasets.

Figures (11)

• Figure 1: Visual architecture-based comparison of: classical ResNet feature extraction mechanism (a), Inception.v3 based feature extraction (b), and the proposed U-IncNet (c).
• Figure 2: The proposed SGAN learning architecture
• Figure 3: Overview of our proposed collaboration-based pipeline that consists of two generators where the first one $\mathbf{G}_{d}$ removes rain strikes/drops from input images while the second generator $\mathbf{G}_{r}$ corrects the visual artifacts that may be caused by the first generator.
• Figure 4: Visual examples of the realistic dataset: $1^{st}$ and $3^{rd}$ columns show snowy images while $2^{nd}$ and $4^{th}$ show their corresponding estimated ground-truths.
• Figure 5: Visual examples of the snow removal results extracted from synthetic snowy videos: $1^{st}$ row shows several snowy frames, $2^{nd}$ row shows the original ground-truth, and $3^{rd}$ row shows the estimated ground-truths using our video based snow removal