RainyGS: Efficient Rain Synthesis with Physically-Based Gaussian Splatting

TL;DR

RainyGS tackles the challenge of photorealistic rain synthesis in open-world scenes by integrating physically-based raindrop and shallow-water simulations with fast 3D Gaussian Splatting rendering. The approach reconstructs scenes via PGSR and GOF-derived geometry, prepares auxiliary maps, and uses a height-map–driven SWEs model to simulate rain on surfaces, followed by a reflection-aware rasterization that combines refraction, specular highlights, Fresnel effects, and raindrops in screen space. Key contributions include adapting shallow-water dynamics to 3DGS for real-time rain effects, employing screen-space ray tracing to enhance reflections on a rasterization pipeline, and delivering an interactive tool with complete 3D and time annotations for robust rain synthesis in diverse real-world scenes. RainyGS achieves over 30 fps while preserving 3D consistency and physical realism, offering a scalable solution for data augmentation, AR/VR, gaming, and autonomous driving applications.

Abstract

We consider the problem of adding dynamic rain effects to in-the-wild scenes in a physically-correct manner. Recent advances in scene modeling have made significant progress, with NeRF and 3DGS techniques emerging as powerful tools for reconstructing complex scenes. However, while effective for novel view synthesis, these methods typically struggle with challenging scene editing tasks, such as physics-based rain simulation. In contrast, traditional physics-based simulations can generate realistic rain effects, such as raindrops and splashes, but they often rely on skilled artists to carefully set up high-fidelity scenes. This process lacks flexibility and scalability, limiting its applicability to broader, open-world environments. In this work, we introduce RainyGS, a novel approach that leverages the strengths of both physics-based modeling and 3DGS to generate photorealistic, dynamic rain effects in open-world scenes with physical accuracy. At the core of our method is the integration of physically-based raindrop and shallow water simulation techniques within the fast 3DGS rendering framework, enabling realistic and efficient simulations of raindrop behavior, splashes, and reflections. Our method supports synthesizing rain effects at over 30 fps, offering users flexible control over rain intensity -- from light drizzles to heavy downpours. We demonstrate that RainyGS performs effectively for both real-world outdoor scenes and large-scale driving scenarios, delivering more photorealistic and physically-accurate rain effects compared to state-of-the-art methods. Project page can be found at https://pku-vcl-geometry.github.io/RainyGS/

Figures (8)

• Figure 1: Using multi-view images as input (a), RainyGS constructs 3D scenes with physically-based Gaussian Splatting techniques to enable efficient and photorealistic rain synthesis (b). Our approach provides users with flexible control over rain intensity, from light drizzle to heavy downpour (c), achieving high-quality, realistic rain effects in a computationally efficient manner. Zoom in for better visual effects.
• Figure 2: The pipeline of RainyGS. Taken multi-view images as input, we apply PGSR chen2024pgsr to recover both the appearance and geometry of the scene. Next, we generate the height map and use shallow-water simulation techniques to synthesize realistic dynamic water accumulation on the ground, incorporating waves and splashes. For rendering novel views, we prepare auxiliary maps, including appearance, depth, and motion maps, to synthesize rain streaks, reflections, and refraction effects using efficient screen-based ray tracing techniques. Finally, we combine all these elements to present realistic dynamic rainy effects, as shown on the right.
• Figure 3: We demonstrate the effects of each proposed module. Starting with a user-specific view (a), we first use the height map to synthesize rainwater and add specular reflection effects (b) using screen-based ray tracing. Next, we incorporate highlight (c) and refraction effects (d). We also apply the Fresnel term to adjust the rainy tones, making them darker and more realistic (e). Finally, we compose the scene with flying streaks to present the final rainy effects (f).
• Figure 4: Rain synthesis results at the same timestep from multiple views. Runway-V2V fails to maintain 3D consistency, while Rain Motion and Instruct-GS2GS are unable to generate realistic rain streaks and puddles. RainyGS maintains 3D consistency and produces realistic rain effects.
• Figure 5: Rain synthesis results at the same viewpoint from different timesteps are shown. Similarly, Rain Motion generates unrealistic and random rain droplets without hydrops. Runway-V2V produces almost static rain content. In contrast, RainyGS generates realistic time-evolving hydrops and puddles.