ClimateGS: Real-Time Climate Simulation with 3D Gaussian Style Transfer

TL;DR

ClimateGS addresses real time climate testing for autonomous systems by enabling photorealistic climate effects in 3D Gaussian scenes. It introduces a linear color space style transfer of spherical harmonic coefficients, a joint training strategy to stabilize learning and preserve details, and a real time physics based rendering pipeline for climate effects. The method leverages 3D Gaussian Splatting with a Gaussian Color Mapping module to transfer style and render smog, floods and snow efficiently. Experiments on MipNeRF360, Tanks and Temples, and Waymo driving scenes show competitive visual quality with far faster rendering, demonstrating practical applicability for interactive autonomous driving simulation. Overall, ClimateGS enables robust, interactive climate testing by combining fast 3D Gaussian style transfer with physics based climate rendering.

Abstract

Adverse climate conditions pose significant challenges for autonomous systems, demanding reliable perception and decision-making across diverse environments. To better simulate these conditions, physically-based NeRF rendering methods have been explored for their ability to generate realistic scene representations. However, these methods suffer from slow rendering speeds and long preprocessing times, making them impractical for real-time testing and user interaction. This paper presents ClimateGS, a novel framework integrating 3D Gaussian representations with physical simulation to enable real-time climate effects rendering. The novelty of this work is threefold: 1) developing a linear transformation for 3D Gaussian photorealistic style transfer, enabling direct modification of spherical harmonics across bands for efficient and consistent style adaptation; 2) developing a joint training strategy for 3D style transfer, combining supervised and self-supervised learning to accelerate convergence while preserving original scene details; 3) developing a real-time rendering method for climate simulation, integrating physics-based effects with 3D Gaussian to achieve efficient and realistic rendering. We evaluate ClimateGS on MipNeRF360 and Tanks and Temples, demonstrating real-time rendering with comparable or superior visual quality to SOTA 2D/3D methods, making it suitable for interactive applications.

Figures (16)

• Figure 1: Illustration of ClimateGS Pipeline. Our method consists of three stages: (a) Multi-view image-based reconstruction of the 3D scene; (b) Style transfer applied to the reconstructed scene for subsequent climate simulation (\ref{['sec:style_transfer', 'sec:training_strategy']}); (c) Physics-based climate simulation using a real-time renderer (\ref{['sec:climate_simulation']}). This process enables the generation of diverse climate simulations.
• Figure 2: Overview of Style Transfer Training Framework.
• Figure 3: Illustration of GSCM. The GSCM module consists of three color projection matrices ($\Lambda$, $\mathbf{P}$, $\mathbf{Q}$) and a transformation matrix $\mathbf{T}$, mapping 3D Gaussian color $sh_j$ from a unified space to the color space defined by the style image, and then restore it to the spherical harmonic space using the inverse of matrix $\Lambda$.
• Figure 4: Procedure of Snow Sampling. (a) demonstrates our method produces more accurate results compared to $\alpha$-blending; (b) and (c) show that on real-world data, our approach reduces erroneous floaters and provides more accurate depth estimation.
• Figure 5: Gaussian Normal Sampling Process.