Rethinking Rainy 3D Scene Reconstruction via Perspective Transforming and Brightness Tuning

TL;DR

The paper tackles rain-induced degradation in multi-view 3D reconstruction by introducing OmniRain3D, a dataset that captures perspective heterogeneity and brightness dynamicity, and REVR-GSNet, an end-to-end framework that jointly enhances brightness, eliminates rain, and reconstructs a clean radiance field. The method combines Recursive Brightness Enhancement, Gaussian Primitives Optimization, and GS-guided Rain Elimination in a closed-loop, leveraging a differentiable 3D Gaussian Splatting representation. Key contributions include the physically grounded rain model, the perspective-aware data synthesis pipeline, and an end-to-end architecture that outperforms state-of-the-art baselines on synthetic and real rainy data. This work advances robust rainy 3D scene reconstruction and provides a realistic benchmark for future research in deraining and 3D reconstruction under adverse weather.

Abstract

Rain degrades the visual quality of multi-view images, which are essential for 3D scene reconstruction, resulting in inaccurate and incomplete reconstruction results. Existing datasets often overlook two critical characteristics of real rainy 3D scenes: the viewpoint-dependent variation in the appearance of rain streaks caused by their projection onto 2D images, and the reduction in ambient brightness resulting from cloud coverage during rainfall. To improve data realism, we construct a new dataset named OmniRain3D that incorporates perspective heterogeneity and brightness dynamicity, enabling more faithful simulation of rain degradation in 3D scenes. Based on this dataset, we propose an end-to-end reconstruction framework named REVR-GSNet (Rain Elimination and Visibility Recovery for 3D Gaussian Splatting). Specifically, REVR-GSNet integrates recursive brightness enhancement, Gaussian primitive optimization, and GS-guided rain elimination into a unified architecture through joint alternating optimization, achieving high-fidelity reconstruction of clean 3D scenes from rain-degraded inputs. Extensive experiments show the effectiveness of our dataset and method. Our dataset and method provide a foundation for future research on multi-view image deraining and rainy 3D scene reconstruction.

Figures (7)

• Figure 1: Illustration of key characteristics in real rainy scenes. (a) Perspective Heterogeneity: Rain streaks vary in appearance across both vertical and horizontal directions, as shown in real observations. (b) Brightness Dynamicity: An increase in rainfall is often accompanied by a decrease in ambient brightness, as confirmed by images ranging from sunny to heavy rain conditions.
• Figure 2: Overview of the data construction pipeline. Camera viewpoint extraction from clean background images. Rain masks are generated by rendering the 3D rain model from the same viewpoint as the background image. The 3D rain model is controlled by parameters such as rain density. A brightness–rain density mapping function adjusts the brightness of background images under different rainfall levels. Finally, brightness-adjusted backgrounds are combined with rain masks to synthesize dynamically rain-degraded images, reducing the domain gap to real rainy scenes.
• Figure 3: Overview of the REVR-GSNet framework. The RBE progressively improves the brightness of multi-view rainy images. The GPO constructs and optimizes a 3D Gaussian representation using enhanced images and camera poses via differentiable rendering. The GRE fuses rendered and enhanced images to remove rain streaks through a residual recursive network, while also feeding back to refine 3D reconstruction. These components employ a joint alternating optimization strategy that enhances brightness, removes rain streaks, and improves 3D scene reconstruction quality.
• Figure 4: Qualitative comparison of REVR-GSNet and other baselines on selected rainy scenes from the OmniRain3D dataset.
• Figure 5: The visualization compares the rendering quality of REVR-GSNet and baselines on raindrop scenes.