Pushing Rendering Boundaries: Hard Gaussian Splatting

TL;DR

This work proposes Hard Gaussian Splatting, dubbed HGS, which considers multi-view significant positional gradients and rendering errors to grow hard Gaussians that fill the gaps of classical Gaussian Splatting on 3D scenes, thus achieving superior NVS results.

Abstract

3D Gaussian Splatting (3DGS) has demonstrated impressive Novel View Synthesis (NVS) results in a real-time rendering manner. During training, it relies heavily on the average magnitude of view-space positional gradients to grow Gaussians to reduce rendering loss. However, this average operation smooths the positional gradients from different viewpoints and rendering errors from different pixels, hindering the growth and optimization of many defective Gaussians. This leads to strong spurious artifacts in some areas. To address this problem, we propose Hard Gaussian Splatting, dubbed HGS, which considers multi-view significant positional gradients and rendering errors to grow hard Gaussians that fill the gaps of classical Gaussian Splatting on 3D scenes, thus achieving superior NVS results. In detail, we present positional gradient driven HGS, which leverages multi-view significant positional gradients to uncover hard Gaussians. Moreover, we propose rendering error guided HGS, which identifies noticeable pixel rendering errors and potentially over-large Gaussians to jointly mine hard Gaussians. By growing and optimizing these hard Gaussians, our method helps to resolve blurring and needle-like artifacts. Experiments on various datasets demonstrate that our method achieves state-of-the-art rendering quality while maintaining real-time efficiency.

Figures (7)

• Figure 1: Illustration of the Gaussian growing criterion in 3DGS kerbl20233d.$\boldsymbol{g}_{i,j}$ denotes the positional gradient of Gaussian $\mathcal{G}_i$ under viewpoint $j$. $L_j$ denotes the rendering loss under viewpoint $j$, which is computed by averaging the rendering errors $\{e_{\boldsymbol{u}}\}$ of all pixels $\bm{P}$. Larger gradients and rendering errors will be smoothed by the average operation.
• Figure 2: Visual analysis of rendering error guided HGS. (a) Ground truth image. (b) Rendered image by 3DGS. (c) Rendered index of Gaussians with the maximum contribution to each pixel. (d) Projection points of potentially over-large Gaussians. (e) Projection points of hard Gaussians. Potentially over-large Gaussians may indicate false positives for Gaussian growing while hard Gaussians alleviate this (the blue boxes in (d) and (e)).
• Figure 3: Qualitative comparisons on three datasets barron2022mipknapitsch2017tankshedman2018deep. Yellow boxes show challenging areas. Our methods can reconstruct these areas well while other methods fail.
• Figure 4: Qualitative results of ablation study.
• Figure 5: Effect of $\lambda$.