SOGS: Second-Order Anchor for Advanced 3D Gaussian Splatting

TL;DR

SOGS addresses the trade-off in anchor-based 3D Gaussian Splatting between anchor feature size and rendering quality by introducing covariance-based second-order anchors that augment anchor features via cross-feature correlations. It computes the covariance $\boldsymbol{\Sigma}$ and correlation $\boldsymbol{R}$ across anchor dimensions, performs eigendecomposition to obtain top-$M$ eigenvectors $\boldsymbol{P}$, and uses them to enhance anchor representations, enabling high-quality rendering with reduced anchor size. A selective gradient loss $\mathcal{L}_s$ based on Sobel gradient maps dynamically emphasizes hard-to-render regions, further improving texture and geometry fidelity while training with smaller anchors. Extensive experiments across Mip-NeRF360, Tanks&Temples, Deep Blending, and BungeeNeRF show SOGS consistently outperforms Scaffold-GS in PSNR, SSIM, and LPIPS with reduced anchor/model sizes, illustrating practical gains for efficient, high-quality novel-view synthesis.

Abstract

Anchor-based 3D Gaussian splatting (3D-GS) exploits anchor features in 3D Gaussian prediction, which has achieved impressive 3D rendering quality with reduced Gaussian redundancy. On the other hand, it often encounters the dilemma among anchor features, model size, and rendering quality - large anchor features lead to large 3D models and high-quality rendering whereas reducing anchor features degrades Gaussian attribute prediction which leads to clear artifacts in the rendered textures and geometries. We design SOGS, an anchor-based 3D-GS technique that introduces second-order anchors to achieve superior rendering quality and reduced anchor features and model size simultaneously. Specifically, SOGS incorporates covariance-based second-order statistics and correlation across feature dimensions to augment features within each anchor, compensating for the reduced feature size and improving rendering quality effectively. In addition, it introduces a selective gradient loss to enhance the optimization of scene textures and scene geometries, leading to high-quality rendering with small anchor features. Extensive experiments over multiple widely adopted benchmarks show that SOGS achieves superior rendering quality in novel view synthesis with clearly reduced model size.

Figures (6)

• Figure 1: The proposed SOGS can render high-quality textures and geometries and reduce model size simultaneously. The illustrations in (a) and (b) show Rendered Images and Gradient Maps which are produced by Scaffold-GS lu2024scaffold and SOGS, respectively, for the samples 'Stump' and 'Room' from Mip-NeRF360 barron2022mip. The Gradient Maps are extracted with the Sobel operator which highlight the texture and geometry of imaged scenes.
• Figure 2: Overview of the proposed SOGS. Initialized from point clouds, each anchor stores an anchor feature vector $\boldsymbol{f^a} \in \mathbb{R}^{D}$. Putting $\boldsymbol{f^a}$ along $D$ dimensions as $D$ variables and all anchors as observed samples, the second-order anchor statistics capturing co-varying relations across $D$ can be computed from the covariance matrix $\boldsymbol{\Sigma}$ across $D$ dimensions. We select the top $M$ eigenvectors $\boldsymbol{P}$ as the most significant co-variation patterns and combine $\boldsymbol{P}$ with $f^a$ to capture anchor-specific textures and geometries for anchor feature augmentation, second-order anchor construction and Gaussian prediction. With the gradient maps of the rendered image and the ground truth, the selective gradient loss guides to learn to render finer textures and geometries with dynamic region selection.
• Figure 3: Visual illustration of the proposed selective gradient loss (SGL). SGL clearly improves the rendering quality by generating finer textures and details. Zoom in for best view.
• Figure 4: Qualitative comparisons of SOGS with 3D-GS kerbl20233d and Scaffold-GS lu2024scaffold. SOGS achieves smaller model size and superior image rendering with much less artifacts but more fine details. The experiments are conducted over multiple indoor and outdoor scenes including 'Garden', 'Flower' and 'Counter' from the Mip-NeRF360 and 'Quebec' from the BungeeNeRF. Zoom in for best view.
• Figure 5: Anchor feature dimension vs SOGS performance: The performance of SOGS varies with the feature dimension $D$ in each anchor, where increasing $D$ improves SOGS consistently together with the increased model size and computational cost. The graph shows the PSNR of the scene 'Bicycle' from MipNeRF360.