Real-time Global Illumination for Dynamic 3D Gaussian Scenes

TL;DR

The paper tackles the challenge of real-time global illumination for dynamic scenes composed of 3D Gaussian primitives and meshes. It introduces a Gaussian-specific Light Transport Equation, a compound stochastic ray-tracing core, and an optimized Gaussian rasterizer, integrated into a practical RTGI pipeline with direct and indirect illumination, a two-level radiance cache, and glossy reflection handling. Key contributions include the LTE formulation for 3D Gaussians, fast stochastic tracing with proxy geometry, forward rasterization, and a cohesive dynamic GI workflow validated at over 40 fps on complex scenes with both Gaussians and meshes. This work advances real-time relighting and dynamic lighting for Gaussian-based scenes, enabling more realistic rendering and bridging rendering techniques with real-world Gaussian reconstructions.

Abstract

We present a real-time global illumination approach along with a pipeline for dynamic 3D Gaussian models and meshes. Building on a formulated surface light transport model for 3D Gaussians, we address key performance challenges with a fast compound stochastic ray-tracing algorithm and an optimized 3D Gaussian rasterizer. Our pipeline integrates multiple real-time techniques to accelerate performance and achieve high-quality lighting effects. Our approach enables real-time rendering of dynamic scenes with interactively editable materials and dynamic lighting of diverse multi-lights settings, capturing mutual multi-bounce light transport (indirect illumination) between 3D Gaussians and mesh. Additionally, we present a real-time renderer with an interactive user interface, validating our approach and demonstrating its practicality and high efficiency with over 40 fps in scenes including both 3D Gaussians and mesh. Furthermore, our work highlights the potential of 3D Gaussians in real-time applications with dynamic lighting, offering insights into performance and optimization.

Figures (15)

• Figure 1: Our RTGI pipeline for 3D Gaussian models. The pipeline uses light sampling from light grid and shadow ray-tracing for direct diffuse lighting. The two-level-cache including the hash grid and screen probes maintains indirect diffuse lighting. Shading ray-tracing that queries intersection properties is used to update the cache and render glossy reflectance.
• Figure 2: Illustration of a shading ray traced in our algorithm. Hardware ray tracing does not guarantee a specific anyhit invocation order along the ray; we illustrate one possible permutation of intersections. First, the GPU reports an intersection with the proxy geometry at a, but the hit is rejected as its opacity $A_\mathbf{v}(a)$ is lower than the random threshold $x_1$. Then, the GPU reports intersection d. The orange Gaussian passes the opacity test, culling subsequent intersections (e, f). Finally, b is accepted, culling c. As the closest hit, b's features are returned as the trace result.
• Figure 3: Ray-traced hit feature (albedo) from nvidia3DGSRT2024 and our stochastic ray tracing, respectively. $A_\mathbf{v}(r)$ in stochastic ray tracing is replaced with Gaussian Max-response for consistency. The initial random value is scaled by 1 or 0.2. We visualize the mean of the unscaled stochastic ray-trace result across multiple frames, which can be identical to nvidia3DGSRT2024. We also show the performance gain in ray throughput.
• Figure 4: At each frame, 2D hexagons are generated from 3D Gaussians for hardware rasterization, aligned with their projected distributions. Depth values ($D_x, D_y, D_c$) are extracted from the 3D Gaussian Max-response positions along camera rays and stored as vertex attributes for hardware interpolation.
• Figure 5: Reconstructed normals of the flat underside of a 3D Gaussian model from depths rendered with constant Gaussian depths (left) and our approach (right). We mitigate the artifacts at the edges of hexagons, by using linear gradients for depths on each hexagon instead of constants.