Not Just Streaks: Towards Ground Truth for Single Image Deraining

TL;DR

The paper addresses single-image deraining and the lack of real ground-truth data, highlighting the sim2real gap introduced by synthetic rain. It introduces GT-RAIN, a large-scale real paired dataset of rainy and clean frames captured from YouTube live streams within short time windows to minimize non-rain variations, comprising 31,524 pairs across 101 videos. It then proposes a rain-robust representation learned via a contrastive InfoNCE objective between rainy and corresponding embeddings ($z_I$ and $z_J$) together with MS-SSIM and $\ell_1$ reconstruction, realized by a deformable-convolution network. Empirical results on GT-RAIN and real internet rain images show substantial improvements over state-of-the-art methods, validating the real paired data approach and suggesting avenues for blending synthetic and real data and refining rain simulators.

Abstract

We propose a large-scale dataset of real-world rainy and clean image pairs and a method to remove degradations, induced by rain streaks and rain accumulation, from the image. As there exists no real-world dataset for deraining, current state-of-the-art methods rely on synthetic data and thus are limited by the sim2real domain gap; moreover, rigorous evaluation remains a challenge due to the absence of a real paired dataset. We fill this gap by collecting a real paired deraining dataset through meticulous control of non-rain variations. Our dataset enables paired training and quantitative evaluation for diverse real-world rain phenomena (e.g. rain streaks and rain accumulation). To learn a representation robust to rain phenomena, we propose a deep neural network that reconstructs the underlying scene by minimizing a rain-robust loss between rainy and clean images. Extensive experiments demonstrate that our model outperforms the state-of-the-art deraining methods on real rainy images under various conditions. Project website: https://visual.ee.ucla.edu/gt_rain.htm/.

Figures (7)

• Figure 1: The points above depict datasets and their corresponding outputs from models trained on them. These outputs come from a real rain image from the Internet. Our opinion* is that GT-RAIN can be the right dataset for the deraining community to use because it has a smaller domain gap to the ideal ground truth. * Why an asterisk? The asterisk emphasizes that this is an "opinion". It is impossible to quantify the domain gap because collecting true real data is infeasible. To date, deraining is largely a viewer's imagination of what the derained scene should look like. Therefore, we present the derained images above and leave it to the viewer to judge the gap. Additionally, GT-RAIN can be used in complement with the litany of synthetic datasets fu2017removinghu2019depthli2019heavyli2016rainyang2017deepzhang2018densityzhang2019image, as illustrated in \ref{['tab:finetune_results']}.
• Figure 2: We collect the a real paired deraining dataset by rigorously controlling the environmental variations. First, we remove heavily degraded videos such as scenes without proper exposure, noise, or water droplets on the lens. Next, we carefully choose the rainy and clean frames as close as possible in time to mitigate illumination shifts before cropping to remove large movement. Lastly, we correct for small camera motion (due to strong wind) using SIFT lowe2004sift and RANSAC fischler1981random and perform elastic image registration thirion1998imagevercauteren2009diffeomorphic by estimating the displacement field when necessary.
• Figure 3: Our proposed dataset contains diverse rainy images collected across the world. We illustrate several representative image pairs with various rain streak appearances and rain accumulation strengths at different geographic locations.
• Figure 4: By minimizing a rain-robust objective, our model learns robust features for reconstruction. When training, a shared-weight encoder is used to extract features from rainy and ground-truth images. These features are then evaluated with the rain-robust loss, where features from a rainy image and its ground-truth are encouraged to be similar. Learned features from the rainy images are also fed into a decoder to reconstruct the ground-truth images with MS-SSIM and $\ell1$ loss functions.
• Figure 5: Our model simultaneously removes rain streaks and rain accumulation, while the existing models fail to generalize to real-world data. The red arrows highlight the difference between the proposed and existing methods on the GT-RAIN test set (zoom for details, PSNR and SSIM scores are listed below the images).