MaterialRefGS: Reflective Gaussian Splatting with Multi-view Consistent Material Inference

TL;DR

MaterialRefGS tackles reflective scene modeling for novel-view synthesis by enforcing multi-view consistent material inference on Gaussian Splatting representations, augmented with a reflection-strength prior derived from photometric variation. It introduces differentiable environment modeling via 2D Gaussian ray tracing to capture indirect illumination under inter-object occlusions, and aligns material maps across views to stabilize illumination decomposition. The method achieves state-of-the-art rendering quality and accurate geometry on synthetic and real datasets, demonstrating improved generalization to complex reflective scenes. This approach enables more physically grounded rendering with efficient splatting-based pipelines.

Abstract

Modeling reflections from 2D images is essential for photorealistic rendering and novel view synthesis. Recent approaches enhance Gaussian primitives with reflection-related material attributes to enable physically based rendering (PBR) with Gaussian Splatting. However, the material inference often lacks sufficient constraints, especially under limited environment modeling, resulting in illumination aliasing and reduced generalization. In this work, we revisit the problem from a multi-view perspective and show that multi-view consistent material inference with more physically-based environment modeling is key to learning accurate reflections with Gaussian Splatting. To this end, we enforce 2D Gaussians to produce multi-view consistent material maps during deferred shading. We also track photometric variations across views to identify highly reflective regions, which serve as strong priors for reflection strength terms. To handle indirect illumination caused by inter-object occlusions, we further introduce an environment modeling strategy through ray tracing with 2DGS, enabling photorealistic rendering of indirect radiance. Experiments on widely used benchmarks show that our method faithfully recovers both illumination and geometry, achieving state-of-the-art rendering quality in novel views synthesis.

Figures (9)

• Figure 1: We highlight our novel view synthesis results on real-world scenes with complex reflection effects. Our MaterialRefGS outperforms the state-of-the-art methods in producing photorealistic renderings, disentangling physical materials, and recovering accurate scene geometry.
• Figure 2: Overview of our method. We learn illumination decomposition by imposing multi-view material consistency constraint and reflection strength prior supervision on the rasterized material maps (Sec. \ref{['sec:regularization']}). To facilitate this process, we introduce an environment modeling strategy through ray tracing with 2DGS, which effectively captures photorealistic incident lighting effects (Sec. \ref{['sec:raytracing']}).
• Figure 3: Illustration of computing reflection strength priors.
• Figure 4: Visual comparisons on Synthetic Datasets. Our method successfully recovers fine-grained reflections on the helmet, as well as inter-reflection effects on the teapot.
• Figure 5: Visual comparisons in real-world Ref-Real verbin2022refnerf dataset. Our method accurately reconstructs reflection textures from surrounding environments on highly reflective surfaces.