Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting

TL;DR

Spec-Gaussian tackles the limitations of 3D Gaussian Splatting in modeling specular and anisotropic appearance by introducing anisotropic spherical Gaussian (ASG) appearance fields and a coarse-to-fine training strategy. It replaces low-frequency SH with ASG for per-Gaussian view-dependent appearance, uses a compact MLP to predict ASG parameters, and decouples diffuse and specular colors with a decoding network to capture high-frequency detail. The method adds anchor-based Gaussian splatting to reduce storage and accelerate rendering, and demonstrates state-of-the-art rendering quality on both synthetic and real-world datasets, while maintaining real-time performance. The work broadens the applicability of 3D-GS to complex materials and anisotropic scenes, and provides a practical framework for efficient, high-fidelity neural splatting.

Abstract

The recent advancements in 3D Gaussian splatting (3D-GS) have not only facilitated real-time rendering through modern GPU rasterization pipelines but have also attained state-of-the-art rendering quality. Nevertheless, despite its exceptional rendering quality and performance on standard datasets, 3D-GS frequently encounters difficulties in accurately modeling specular and anisotropic components. This issue stems from the limited ability of spherical harmonics (SH) to represent high-frequency information. To overcome this challenge, we introduce Spec-Gaussian, an approach that utilizes an anisotropic spherical Gaussian (ASG) appearance field instead of SH for modeling the view-dependent appearance of each 3D Gaussian. Additionally, we have developed a coarse-to-fine training strategy to improve learning efficiency and eliminate floaters caused by overfitting in real-world scenes. Our experimental results demonstrate that our method surpasses existing approaches in terms of rendering quality. Thanks to ASG, we have significantly improved the ability of 3D-GS to model scenes with specular and anisotropic components without increasing the number of 3D Gaussians. This improvement extends the applicability of 3D GS to handle intricate scenarios with specular and anisotropic surfaces. Project page is https://ingra14m.github.io/Spec-Gaussian-website/.

Figures (19)

• Figure 1: Our method not only achieves real-time rendering but also significantly enhances the capability of 3D-GS to model scenes with specular and anisotropic components. Key to this enhanced performance is our use of ASG appearance field to model the appearance of each 3D Gaussian, which results in substantial improvements in rendering quality for both complex and general scenes.
• Figure 2: Pipeline of Spec-Gaussian. The optimization process begins with SfM points derived from COLMAP or generated randomly, serving as the initial state for the 3D Gaussians. To address the limitations of low-order SH and pure MLP in modeling high-frequency information, we additionally employ ASG in conjunction with a feature decoupling MLP to model the view-dependent appearance of each 3D Gaussian. Then, 3D Gaussians with opacity $\sigma > 0$ are rendered through a differentiable Gaussian rasterization pipeline, effectively capturing specular highlights and anisotropy in the scene.
• Figure 3: Quantitative Comparison on anisotropic synthetic dataset.
• Figure 4: Using a coarse-to-fine strategy, our approach can eliminate the floaters without increasing the number of GS.
• Figure 5: Visualization on NeRF dataset. Our method has achieved specular highlights modeling, which other 3D-GS-based methods fail to accomplish, while maintaining fast rendering speed.