TRG-Net: An Interpretable and Controllable Rain Generator

TL;DR

This work tackles the scarcity and variability of real rainy data by introducing TRG-Net, a rain generator that intrinsically encodes fundamental rain factors such as orientation $\theta$, length $s_l$, width $s_w$, shape $\boldsymbol{\alpha}$, and sparsity $\tau$ within a transformable convolution framework. It統ercedes a three-part backbone comprising a learnable rain kernel dictionary, a rotatable rain map, and a merging network, with rain generation expressed as $\mathcal{X}(\theta, s_l, s_w, \boldsymbol{\alpha}, \tau) = \mathrm{MerNet}(\mathcal{C}(\theta, s_l, s_w, \boldsymbol{\alpha}) \otimes \mathcal{M}(\tau, \theta), \mathcal{B})$, enabling controllable and data-driven modeling of rain patterns without explicit factor labels. A rotatable TV regularizer $L_{rotTV} = \| D(\theta) \otimes \mathcal{R} \|_1$ further aligns variation analysis with rain streak orientation. Empirical results show TRG-Net achieves higher rain realism (lower FID/KID), provides diverse and physically meaningful rain samples for unpaired generation and deraining augmentation, and improves cross-domain generalization, outperforming state-of-the-art GAN-based rain generators. The approach offers interpretable control over rain factors and demonstrates practical impact for data augmentation in rainy image processing tasks, with potential applicability to broader vision problems requiring transformable convolution and orientation-aware regularization.

Abstract

Exploring and modeling rain generation mechanism is critical for augmenting paired data to ease training of rainy image processing models. Against this task, this study proposes a novel deep learning based rain generator, which fully takes the physical generation mechanism underlying rains into consideration and well encodes the learning of the fundamental rain factors (i.e., shape, orientation, length, width and sparsity) explicitly into the deep network. Its significance lies in that the generator not only elaborately design essential elements of the rain to simulate expected rains, like conventional artificial strategies, but also finely adapt to complicated and diverse practical rainy images, like deep learning methods. By rationally adopting filter parameterization technique, we first time achieve a deep network that is finely controllable with respect to rain factors and able to learn the distribution of these factors purely from data. Our unpaired generation experiments demonstrate that the rain generated by the proposed rain generator is not only of higher quality, but also more effective for deraining and downstream tasks compared to current state-of-the-art rain generation methods. Besides, the paired data augmentation experiments, including both in-distribution and out-of-distribution (OOD), further validate the diversity of samples generated by our model for in-distribution deraining and OOD generalization tasks.

Figures (9)

• Figure 1: The pipeline for artificial rainy image synthesis based on physical rendering. It mainly contains three parts: (a) rain kernel model, (b) rain map model, and (c) merging model.
• Figure 2: The orientation distribution (left figure) extracted from Rain100L by the proposed TRG-Net, which enables to generate rains with required orientation by specifying the orientation degree $\theta$. Here, we generate the rains with orientations of $0^{\circ}$, $30^{\circ}$ and $60^{\circ}$, respectively.
• Figure 3: Transform the convolution output by the transformation of the convolution filter, which is achieved by functional transforms on the underlying 2D basis functions. $\varphi(\theta,s_l,s_w)$ and $\varphi(\hat{\theta},\hat{s}_l,\hat{s}_w)$ are the basis function set (defined in Eqs. (\ref{['basis set']}) and (\ref{['improved_Bases']})) with different transformation parameters. $C(\theta,s_l,s_w)$ and $C(\hat{\theta},\hat{s}_l,\hat{s}_w)$ represent two convolution kernels (defined in Eq. (\ref{['discretization']})) that share the same combination coefficients $\bm{w}$, but have different transformation parameters. $R$ and $\hat{R}$ denote the convolution outputs of rain map $M$ with $C(\theta,s_l,s_w)$ and $C(\hat{\theta},\hat{s}_l,\hat{s}_w)$, respectively.
• Figure 4: The proposed transformable rainy image generation network (TRG-Net), which takes the proposed rain model (\ref{['rain_model']}) as the backbone. Similar to the conventional artificial rainy image synthesis framework as shown in Fig. \ref{['phy_quan']}, it also consists of three parts: (a) the rain kernel model (Sec. \ref{['SecKernel']}), (b) the rain map model (Sec. \ref{['SecMap']}) and (c) the merging model (Sec. \ref{['SecMerge']}).
• Figure 5: The differential field of a rain layer along its rain streak orientation is significantly sparser than those along other orientations. (a) The differential field of the rain $R(0^{\circ})$ with $0^{\circ}$ orientation along the $0^{\circ}$ orientation is sparse. (b) The differential field of the rain $R(45^{\circ})$ with $45^{\circ}$ orientation along the $45^{\circ}$ orientation is also sparse. (c) The differential field of the rain $R(45^{\circ})$ with $45^{\circ}$ orientation along the $0^{\circ}$ orientation is not sparse.