RainyScape: Unsupervised Rainy Scene Reconstruction using Decoupled Neural Rendering

TL;DR

RainyScape addresses multi-view rainy scene reconstruction without supervision by decoupling rain streaks from the scene using a neural rendering prior and a learnable rain-embedding-based predictor. It introduces an adaptive gradient rotation loss to separate rain from high-frequency details and trains via alternating optimization between a neural rendering module and rain embeddings, with Langevin Monte Carlo refinement. The framework is validated on NeRF and 3D Gaussian Splatting backbones, achieving state-of-the-art deraining performance and new Maya-based rainy datasets for realistic multi-view rain. It offers a general, renderer-agnostic approach to rain mitigation that enables rain-free novel-view rendering in challenging weather scenarios.

Abstract

We propose RainyScape, an unsupervised framework for reconstructing clean scenes from a collection of multi-view rainy images. RainyScape consists of two main modules: a neural rendering module and a rain-prediction module that incorporates a predictor network and a learnable latent embedding that captures the rain characteristics of the scene. Specifically, based on the spectral bias property of neural networks, we first optimize the neural rendering pipeline to obtain a low-frequency scene representation. Subsequently, we jointly optimize the two modules, driven by the proposed adaptive direction-sensitive gradient-based reconstruction loss, which encourages the network to distinguish between scene details and rain streaks, facilitating the propagation of gradients to the relevant components. Extensive experiments on both the classic neural radiance field and the recently proposed 3D Gaussian splatting demonstrate the superiority of our method in effectively eliminating rain streaks and rendering clean images, achieving state-of-the-art performance. The constructed high-quality dataset and source code will be publicly available.

Figures (10)

• Figure 1: Overview of the proposed RainyScape framework, which can reconstruct a rain-free scene from a set of multi-view rainy images in an unsupervised fashion. Based on the NeRF architecture, the rendering module takes ray positions and view directions as input to estimate color and density values. Rain characteristics are modeled using scene state vectors $\textbf{s}$, viewpoint state vectors $v^i$, and camera parameters $p^i$. The combined rain embedding is processed through an MLP to obtain latent space representations, which are then fed into a CNN predictor to produce a rain map. The framework is trained using unsupervised losses that facilitate the decoupling of high-frequency scene details and rain streaks, yielding a rain-free neural radiance field.
• Figure 2: Illustration of neural rendering prior: From low-frequency scene representation to high-frequency detail restoration and rain preservation. The numbers represent the training epoch, the PSNR (dB) of the rendered image compared with rain-free ground truth images, and the PSNR (dB) of the rendered image compared with the rainy image.
• Figure 3: Leveraging directional sensitivity and gradient orientation for unsupervised rainy scene reconstruction. (a) Directional sensitivity of gradient magnitude differences between rainy and rain-free images. Gradients perpendicular to the rain direction exhibit higher discriminative power compared to those along the rain direction. (b) Distribution of gradient orientation in the residual map $\textbf{I} - \textbf{B}_l$. The residual map contains gradients in all directions (green dashed lines), with a dominant orientation perpendicular to the rain direction (red dashed lines) due to the presence of rain streaks.
• Figure 4: Scenes and names of the proposed dataset.
• Figure 5: Visual comparison of different methods on rainy scenes. Each scene shows rainy images, ground truth, rendered results from baseline methods and our approach, error maps, and rain streak images predicted by our method.