Spectral-GS: Taming 3D Gaussian Splatting with Spectral Entropy

TL;DR

The Spectral-GS, based on spectral analysis, introduces 3D shape-aware splitting and 2D view-consistent filtering strategies, enhancing 3DGS’s capability to represent high-frequency details without noticeable artifacts, and achieving high-quality realistic rendering.

Abstract

Recently, 3D Gaussian Splatting (3D-GS) has achieved impressive results in novel view synthesis, demonstrating high fidelity and efficiency. However, it easily exhibits needle-like artifacts, especially when increasing the sampling rate. Mip-Splatting tries to remove these artifacts with a 3D smoothing filter for frequency constraints and a 2D Mip filter for approximated supersampling. Unfortunately, it tends to produce over-blurred results, and sometimes needle-like Gaussians still persist. Our spectral analysis of the covariance matrix during optimization and densification reveals that current 3D-GS lacks shape awareness, relying instead on spectral radius and view positional gradients to determine splitting. As a result, needle-like Gaussians with small positional gradients and low spectral entropy fail to split and overfit high-frequency details. Furthermore, both the filters used in 3D-GS and Mip-Splatting reduce the spectral entropy and increase the condition number during zooming in to synthesize novel view, causing view inconsistencies and more pronounced artifacts. Our Spectral-GS, based on spectral analysis, introduces 3D shape-aware splitting and 2D view-consistent filtering strategies, effectively addressing these issues, enhancing 3D-GS's capability to represent high-frequency details without noticeable artifacts, and achieving high-quality photorealistic rendering.

Figures (12)

• Figure 1: Despite its high efficiency in 3D reconstruction, 3D Gaussian Splatting (3D-GS) kerbl20233d suffers from needle-like artifacts (c) due to undersampling or view inconsistency. Recent works like Mip-Splatting mip_gs and Analytic-Splatting liang2024analytic try to eliminate these artifacts. Unfortunately, they still produce needles at high-frequency regions when zooming in, and will also cause over-blurriness (d)(e) since they lack shape awareness of 3D Gaussians. With spectral analysis of the variance matrix, we propose Spectral-GS, which imposes shape constraints on the 3D Gaussians and thus effectively addresses the above issues, generating high-quality photorealistic rendering (b).
• Figure 2: Visualization of Gaussians with the same spectral radius but different shapes. The spectrum of the 3D Gaussian is characterized by $s_1, s_2, s_3$ (top row), while the 2D Gaussian is characterized by $s_1, s_2$ (bottom row). From left to right, as the spectral entropy decreases and the condition number increases, the Gaussians transition from isotropic to anisotropic.
• Figure 3: Illustrations of the optimization and densification of Gaussians in 3D-GS kerbl20233d. Correct Gaussians: When view-positional gradients $\nabla_{\boldsymbol{\mu}_{\text{proj}}}\mathcal{L}$ exceed a certain threshold $\tau_{\text{loss}}$, 3D-GS decides to clone or split based on the Gaussian's spectral radius $\rho\left(\boldsymbol{\Sigma}\right)$. Needle-like Gaussians: However, 3D-GS does not split or clone Gaussians with low spectral entropy but small gradients.
• Figure 4: Illustrations of the condition number variation curve and rendering results when zooming in. We fix the condition number of 3D Gaussians $\kappa\left(\boldsymbol{\Sigma}\right)=144$ during training. Due to view-inconsistency in filtering, the train view still produces satisfactory rendering results (2D Mip Filter train view), but the test view with higher $\frac{f_x^2}{\mu_z^2}$ shows needle-like artifacts (2D Mip Filter test view).
• Figure 5: Overview of Spectral-GS. 3D Gaussian Splatting (3D-GS)kerbl20233d decides whether to split based on the positional gradients and the spectral radius of the covariance matrix without considering the shape of primitives. We propose the 3D shape-aware splitting strategy based on the spectral analysis (3D Split). In screen space, both the EWA filter zwicker2002ewa of 3D-GS which attempts to cover an entire pixel, and the Mip filter of Mip-Splattingmip_gs which approximates supersampling, result in a reduction of spectral entropy when zooming in to synthesize novel view. Our view-consistent filter's kernel is not constant to maintain the spectral entropy consistency (2D Filter).