RainSD: Rain Style Diversification Module for Image Synthesis Enhancement using Feature-Level Style Distribution

TL;DR

This work tackles sensor blockage in autonomous driving caused by heavy rain by generating realistic rainy training data from real-road images. It introduces RainSD, a rainy-style diversification module that augments a TSIT-based image-to-image translator to produce rain-streaked scenes across varied rainfall rates ($10$–$100$ mm/h) while preserving scene content. A synthetic dataset derived from BDD100K is used to benchmark multi-task perception networks (lane detection, driving-area segmentation, and traffic object detection), with RainSD+TSIT showing robustness gains over baselines in many scenarios. The paper also analyzes feature-level style distribution shifts, noting that heavier rain tends to darken outputs and discussing implications for future rainy-image synthesis and model evaluation.

Abstract

Autonomous driving technology nowadays targets to level 4 or beyond, but the researchers are faced with some limitations for developing reliable driving algorithms in diverse challenges. To promote the autonomous vehicles to spread widely, it is important to address safety issues on this technology. Among various safety concerns, the sensor blockage problem by severe weather conditions can be one of the most frequent threats for multi-task learning based perception algorithms during autonomous driving. To handle this problem, the importance of the generation of proper datasets is becoming more significant. In this paper, a synthetic road dataset with sensor blockage generated from real road dataset BDD100K is suggested in the format of BDD100K annotation. Rain streaks for each frame were made by an experimentally established equation and translated utilizing the image-to-image translation network based on style transfer. Using this dataset, the degradation of the diverse multi-task networks for autonomous driving, such as lane detection, driving area segmentation, and traffic object detection, has been thoroughly evaluated and analyzed. The tendency of the performance degradation of deep neural network-based perception systems for autonomous vehicle has been analyzed in depth. Finally, we discuss the limitation and the future directions of the deep neural network-based perception algorithms and autonomous driving dataset generation based on image-to-image translation.

Figures (15)

• Figure 1: The generated images fusing our rainy style diversification (RainSD) module and image-to-image translation model, including the rainfall rate of 20mm/h, 60mm/h, and 100m/h. (a) depicts translations from clear daytime to rainy daytime, (b) illustrates transitions from clear day to rainy night, and (c) displays clear night to rainy night transitions. Combined with feature-level style transfer based image-to-image translation model, our proposed method successfully generates fake rainy images with diverse rainfall rate preserving the existing semantic information of the content image.
• Figure 2: The overall structure of the proposed network with our novel RainSD module. Style fusion blocks include FAdaIN style transfer module and FADE ResBlock from 42.jiang2020tsit. The network is consisted of two streams of encoder-decoder structure, which enables the network to learn style distribution of both forward feature and backward feature. A generation process of the synthetic image is highly inspired from 42.jiang2020tsit, and we attempt to vary the styles of the rainy scenes, especially by generating realistic rain streaks based on application of the experimentally obtained equation. Detailed explanation on RainSD module is in Fig.4
• Figure 3: Comparison of the generated images from the representative image-to-image translation models, comprised of MUNIT, DRIT++, and TSIT. Both semantic information and the style information are rarely transferred when using MUNIT and DRIT++, making unintended distortion to the crucial objects in the road scenes. TSIT successfully translates the style information from the rainy images possess to the content image, preserving the important semantic structures.
• Figure 4: The structure of the proposed RainSD module.
• Figure 5: Qualitative results of our novel RainSD module combined with different kinds of image-to-image translation methods. It shows the images generated from clear-daytime (test A, content image) to rainy-daytime (test B, style image). The images depict the results of image synthesis process with regard to the rainfall amounts including 20mm/h, 40mm/h, 60mm/h, 80mm/h, and 100mm/h, upon consideration of realistic calculation 19.jeon2022carla using the method proved by image similarity metrics. MUNIT and DRIT++ fail to deliver the amount of rain streaks in each style image, whereas TSIT successfully conducts style transfer.