3DGS$^2$: Near Second-order Converging 3D Gaussian Splatting

TL;DR

This paper addresses the slow, gradient-based training of 3D Gaussian Splatting (3DGS) by introducing a stochastic local Newton method that decouples kernel attributes and updates them with small, per-attribute Newton steps. By leveraging weak coupling among kernel parameters and spatial relationships among training views, the approach mitigates overshoot with secondary-target losses and achieves near second-order convergence, reducing training iterations by an order of magnitude while preserving or improving rendering quality. Extensive experiments on standard 3DGS datasets show 5–10× faster convergence compared to SGD-based training, highlighting practical speedups from seconds to minutes without sacrificing visual fidelity. The work advances efficient 3D scene reconstruction with Gaussian splatting and offers a scalable framework for higher-order optimization in structured 3D representations.

Abstract

3D Gaussian Splatting (3DGS) has emerged as a mainstream solution for novel view synthesis and 3D reconstruction. By explicitly encoding a 3D scene using a collection of Gaussian kernels, 3DGS achieves high-quality rendering with superior efficiency. As a learning-based approach, 3DGS training has been dealt with the standard stochastic gradient descent (SGD) method, which offers at most linear convergence. Consequently, training often requires tens of minutes, even with GPU acceleration. This paper introduces a (near) second-order convergent training algorithm for 3DGS, leveraging its unique properties. Our approach is inspired by two key observations. First, the attributes of a Gaussian kernel contribute independently to the image-space loss, which endorses isolated and local optimization algorithms. We exploit this by splitting the optimization at the level of individual kernel attributes, analytically constructing small-size Newton systems for each parameter group, and efficiently solving these systems on GPU threads. This achieves Newton-like convergence per training image without relying on the global Hessian. Second, kernels exhibit sparse and structured coupling across input images. This property allows us to effectively utilize spatial information to mitigate overshoot during stochastic training. Our method converges an order faster than standard GPU-based 3DGS training, requiring over $10\times$ fewer iterations while maintaining or surpassing the quality of the compared with the SGD-based 3DGS reconstructions.

Figures (6)

• Figure 1: Improve 3DGS training using parallelized local Newton. We show 3DGS training can be substantially improved by switching from stochastic gradient descent to stochastic Newton. A Gaussian kernel has multiple attributes, and optimizing all the attributes of all the kernels as a whole leads to a very high-dimensional nonlinear problem. We leverage the fact that kernel attributes are weakly coupled and design a local Newton scheme to find the optimal value of each type of kernel parameter. We also exploit the spatial relation among input images to effectively mitigate the overshoot issue in stochastic optimization. As a result, our method is $5\times$ to $10\times$ faster than gradient-based 3DGS training. In the teaser, we show the reconstruction results using the-state-of-the-art 3DGS training algorithms, including vanilla 3DGS kerbl20233d, AdR-Gaussian adr2024, EAGLES EAGLEs2024, Taming Gaussian Taming3DGS as well as our method. Our method produces the result of the highest quality (in terms of SSIM, PSNR, and LPIPS) while using only one-tenth of iterations, making the reconstruction of complex 3D scenes in seconds instead of minutes.
• Figure 2: Hessian visualization. We plot the values of local and global Hessian matrices for one training image. The local Hessian (left) is for all the parameters of a kernel $\bm{x}_k = [\bm{p}_k^\top, \bm{q}_k^\top, \bm{s}_k^\top, \bm{c}_k^\top, \sigma_k]^\top$, while the global Hessian (right) is for all the kernels' parameters. The variation of the matrices across different DOFs suggests (very) weak coupling amount parameters.
• Figure 3: Data preparation. We construct a bounding sphere of the scene containing all the GS kernels. Camera poses corresponding to input training images are projected to the sphere surface, and the spherical distance between them is used as the metric to identify KNNs for training images. The current training image is the primary target while its KNNs are the secondary targets.
• Figure 4: Convergence curves of our method and 3DGS. We report a group of representative convergence plots using the proposed training method (local Newton) and vanilla 3DGS training (GD). The corresponding input training images are also attached next to the curves. We observe a strong second-order convergence using our method compared with gradient-based training in almost all the scenes.
• Figure 5: Training order of kernel attributes. We always train the kernel position before the color information (RGB and opacity). Without a good positioning of the kernel, there is limited space for color-wise optimization. If one chooses to train the kernel color first, the training converges quadratically as well, but to a different local minimum. Meanwhile, the training order of rotation and scaling or opacity and RBG information is not important. The convergence curves are nearly identical as those attributes are de-coupled.