VoD-3DGS: View-opacity-Dependent 3D Gaussian Splatting

TL;DR

VoD-3DGS addresses the challenge of view-dependent lighting in 3D reconstruction by introducing a learnable symmetric opacity matrix per Gaussian, enabling suppression or enhancement of Gaussians based on the viewing direction. The approach combines this view-dependent opacity with density control, an opacity reset mechanism, and a view-consistency loss to stabilize appearance across views, achieving state-of-the-art quality on multiple benchmarks while preserving real-time rendering (>60 FPS). It maintains the explicit Gaussian representation of 3DGS and avoids additional neural networks, offering a lightweight yet effective improvement for specular highlights and dynamic lighting. The method demonstrates strong cross-dataset performance with modest memory overhead, representing a practical advancement for photorealistic view synthesis in real-time applications.

Abstract

Reconstructing a 3D scene from images is challenging due to the different ways light interacts with surfaces depending on the viewer's position and the surface's material. In classical computer graphics, materials can be classified as diffuse or specular, interacting with light differently. The standard 3D Gaussian Splatting model struggles to represent view-dependent content, since it cannot differentiate an object within the scene from the light interacting with its specular surfaces, which produce highlights or reflections. In this paper, we propose to extend the 3D Gaussian Splatting model by introducing an additional symmetric matrix to enhance the opacity representation of each 3D Gaussian. This improvement allows certain Gaussians to be suppressed based on the viewer's perspective, resulting in a more accurate representation of view-dependent reflections and specular highlights without compromising the scene's integrity. By allowing the opacity to be view dependent, our enhanced model achieves state-of-the-art performance on Mip-Nerf, Tanks&Temples, Deep Blending, and Nerf-Synthetic datasets without a significant loss in rendering speed, achieving >60FPS, and only incurring a minimal increase in memory used.

Figures (3)

• Figure 1: [Top] The visualization of the effect of our method in comparison to the standard 3D Gaussian Splatting. We can suppress or boost the impact of Gaussians responsible for modeling specular highlights and reflections for certain views by extending the opacity component of each 3D Gaussian with a symmetric matrix, acting as a learnable view-dependent factor. [Bottom] Qualitative results of our proposed method in comparison to standard 3DGS. When multiplied with the view vector, the symmetric matrix allows us to represent reflections, specular highlights, and even changing lights.
• Figure 2: Comparison of our proposed method to standard 3DGS. Our method boosts the specular response in various scenes, which can be seen on the table center (Garden scene; first row), the water filter (Counter scene; fourth row), the rightmost corner of the table (Room scene; last row), and the left side of the base of the bonsai (Bonsai scene; third row). Our method also allows changing light conditions (white gravel and train tracks in the Train scene; second row).
• Figure 3: Our method boosts the reflection in the drums (Drums scene), on the hotdog plate (Hotdog scene), and the reflections of the red and gray spheres in the copper materials (Materials scene) of the NeRF-Synthetic nerf dataset, achieving state-of-the-art results.