Steepest Descent Density Control for Compact 3D Gaussian Splatting

TL;DR

3D Gaussian Splatting (3DGS) can accumulate redundant Gaussian points during densification, hindering efficiency and scalability. This work introduces a theory-driven Steepest Density Control (SDC) that uses a per-point splitting matrix to decide when and how to densify, with offspring placed along the least-eigenvalue direction and opacity halved to preserve local density. The resulting SteepGS system integrates into the 3DGS CUDA kernel and achieves about a 50% reduction in Gaussian count while maintaining rendering quality, improving memory and rendering speed. The approach provides a principled alternative to heuristic densification, enabling more scalable real-time view synthesis on resource-constrained devices.

Abstract

3D Gaussian Splatting (3DGS) has emerged as a powerful technique for real-time, high-resolution novel view synthesis. By representing scenes as a mixture of Gaussian primitives, 3DGS leverages GPU rasterization pipelines for efficient rendering and reconstruction. To optimize scene coverage and capture fine details, 3DGS employs a densification algorithm to generate additional points. However, this process often leads to redundant point clouds, resulting in excessive memory usage, slower performance, and substantial storage demands - posing significant challenges for deployment on resource-constrained devices. To address this limitation, we propose a theoretical framework that demystifies and improves density control in 3DGS. Our analysis reveals that splitting is crucial for escaping saddle points. Through an optimization-theoretic approach, we establish the necessary conditions for densification, determine the minimal number of offspring Gaussians, identify the optimal parameter update direction, and provide an analytical solution for normalizing off-spring opacity. Building on these insights, we introduce SteepGS, incorporating steepest density control, a principled strategy that minimizes loss while maintaining a compact point cloud. SteepGS achieves a ~50% reduction in Gaussian points without compromising rendering quality, significantly enhancing both efficiency and scalability.

Figures (8)

• Figure 1: We theoretically investigate density control in 3DGS. As training via gradient descent progresses, many Gaussian primitives are observed to become stationary while failing to reconstruct the regions they cover (e.g. the cyan-colored blobs in the top-left figure marked with ). From an optimization-theoretic perspective (see figure on the right), we reveal that these primitives are trapped in saddle points, the regions in the loss landscape where gradients are insufficient to further reduce loss, leaving parameters sub-optimal locally. To address this, we introduce SteepGS, which efficiently identifies Gaussian points located in saddle area, splits them into two off-springs, and displaces new primitives along the steepest descent directions. This restores the effectiveness of successive gradient-based updates by escaping the saddle area (e.g. the orange-colored blobs in the top-left figure marked with become optimizable after densification). As shown in the bottom-left visualization, SteepGS achieves a more compact parameterization while preserving the fidelity of fine geometric details.
• Figure 2: Illustrative notation for the splitting process. The updates $\boldsymbol{\vartheta}^{(i)}_j - \boldsymbol{\theta}^{(i)}$ can be decomposed as first taking a mean-field shift $\boldsymbol{\mu}^{(i)}$ and then applying individual updates $\boldsymbol{\delta}^{(i)}_j$. By this decomposition, $\sum w^{(i)}_j \boldsymbol{\delta}^{(i)}_j = \boldsymbol{0}$.
• Figure 3: Illustration of Steepest Density Control. SDC, as the optimal solution to Eq. \ref{['eqn:optim_goal']}, takes the steepest descent on the loss after splitting. Geometrically, it moves two off-spring Gaussians to opposite directions along the smallest eigenvector of the splitting matrix and shrinks the opacity of each Gaussian by 0.5.
• Figure 4: Qualitative Results. We compare our SteepGS with other densification baselines. For each scene, the first row shows the rendered view, while the second row visualizes the error with respect to the ground truth. Key details are highlighted in the blue box.
• Figure 5: Visualization of splitting points. The rendered views are present on the left and the corresponding points to be split are visualized on the right.

Theorems & Definitions (14)

• Theorem 1
• Theorem 2
• proof : Proof of Theorem \ref{['thm:sec_ord_approx']}
• proof : Proof of Theorem \ref{['thm:optim_split']}
• Lemma 3
• proof
• Lemma 4
• proof
• Lemma 5
• proof