EnvGS: Modeling View-Dependent Appearance with Environment Gaussian

TL;DR

EnvGS tackles the challenge of photorealistic, real-time novel view synthesis in scenes with complex reflections by introducing an explicit environment Gaussian primitive set that models view-dependent reflections separate from base geometry. A differentiable Gaussian tracer running on GPU-accelerated ray tracing renders environment Gaussians along reflection directions and supports end-to-end optimization with base Gaussians, enabling high-frequency and near-field reflection detail without relying on distant lighting assumptions. The approach demonstrates superior reflection fidelity across real and synthetic datasets, outperforms real-time baselines, and maintains real-time speeds on high-end GPUs. The work advances practical real-time rendering of complex reflective scenes and paves the way for extending the framework to handle transparent and refractive materials in future research.

Abstract

Reconstructing complex reflections in real-world scenes from 2D images is essential for achieving photorealistic novel view synthesis. Existing methods that utilize environment maps to model reflections from distant lighting often struggle with high-frequency reflection details and fail to account for near-field reflections. In this work, we introduce EnvGS, a novel approach that employs a set of Gaussian primitives as an explicit 3D representation for capturing reflections of environments. These environment Gaussian primitives are incorporated with base Gaussian primitives to model the appearance of the whole scene. To efficiently render these environment Gaussian primitives, we developed a ray-tracing-based renderer that leverages the GPU's RT core for fast rendering. This allows us to jointly optimize our model for high-quality reconstruction while maintaining real-time rendering speeds. Results from multiple real-world and synthetic datasets demonstrate that our method produces significantly more detailed reflections, achieving the best rendering quality in real-time novel view synthesis. The code is available at https://zju3dv.github.io/envgs.

Figures (10)

• Figure 1: Overview of EnvGS. The rendering process begins by rasterizing the base Gaussian to obtain per-pixel normals, base colors, and blending weights. Next, we render the environment Gaussian in the reflection direction using our ray-tracing-based Gaussian renderer to capture the reflection colors. Finally, we combine the reflection and base colors for the final output. We jointly optimize the environment Gaussian and base Gaussian using monocular normals ye2024stablenormal and ground truth images for supervision.
• Figure 2: Visualization of reflection and base color. Our method successfully reconstructs near-field and distant reflections using the environment Gaussian instead of baking into the base color.
• Figure 3: Qualitative comparison on real scenes. Our method significantly improves rendering quality over previous approaches, particularly in producing more detailed reflections. Zoom in for more details.
• Figure 4: Ablation study of proposed components on the Ref-Real dataset verbin2022refnerf. Removing either the monocular normal constraint or the joint optimization of base and environment Gaussians results in noisy geometry and inaccurate reflection reconstruction. The "w/ environment map" variant fails to capture near-field reflections.
• Figure 5: Qualitative comparison on reflective foreground and near-field reflection regions. We also provide visualizations of the foreground and near-field region mask we annotated.