Matrix-free Second-order Optimization of Gaussian Splats with Residual Sampling

TL;DR

The paper tackles slow optimization in explicit 3D Gaussian Splatting by introducing a matrix-free Levenberg–Marquardt framework that exploits Jacobian sparsity with a GPU-accelerated preconditioned conjugate gradient solver. It further accelerates training through view sampling and residual sampling to approximate the normal equations, together with a learning-rate heuristic that eliminates line searches. Empirically, the approach delivers up to 5×–6× speedups over strong baselines while maintaining competitive rendering quality, and uses substantially less memory than Jacobian-cached second-order methods. This work demonstrates the viability of second-order optimization for explicit 3D scene representations and provides practical techniques to scale it on modern GPUs.

Abstract

3D Gaussian Splatting (3DGS) is widely used for novel view synthesis due to its high rendering quality and fast inference time. However, 3DGS predominantly relies on first-order optimizers such as Adam, which leads to long training times. To address this limitation, we propose a novel second-order optimization strategy based on Levenberg-Marquardt (LM) and Conjugate Gradient (CG), which we specifically tailor towards Gaussian Splatting. Our key insight is that the Jacobian in 3DGS exhibits significant sparsity since each Gaussian affects only a limited number of pixels. We exploit this sparsity by proposing a matrix-free and GPU-parallelized LM optimization. To further improve its efficiency, we propose sampling strategies for both the camera views and loss function and, consequently, the normal equation, significantly reducing the computational complexity. In addition, we increase the convergence rate of the second-order approximation by introducing an effective heuristic to determine the learning rate that avoids the expensive computation cost of line search methods. As a result, our method achieves a $3\times$ speedup over standard LM and outperforms Adam by $~6\times$ when the Gaussian count is low while remaining competitive for moderate counts. Project Page: https://vcai.mpi-inf.mpg.de/projects/LM-IS

Figures (7)

• Figure 1: We introduce a matrix-free second-order optimizer for Gaussian splatting. Notably, our dedicated optimizer converges significantly faster than Adam adam and already achieves reasonable renderings after very few seconds of training.
• Figure 2: Overview of our method is given. We start from randomly initialized Gaussians and gradually refine them with Levenberg-Marquardt optimizer. Since dealing with the true Jacobian matrix is costly, we approximate it with a tile-aware sampling algorithm. After we solve the normal equations with approximated Jacobians, we update the parameters using a learning rate heuristic. Note that the Jacobians are never materialized in memory, and the normal equation is solved with our matrix-free algorithm.
• Figure 3: We visualize a normalized $\mathbf{J^\top J}$ matrix for one downsampled training image (a). While elements in the diagonal are common, we compute the dominance ratio as the normalized ratio between the diagonal element over the sum of other elements in the same row (b) and show how only a limited number of parameters lead to a diagonally dominant linear system (c).
• Figure 4: The averaged results of competing methods on the synthetic NeRF dataset are presented. Our method achieves similar quality in a shorter amount of time. We also provide convergence plots on the Lego scene.
• Figure 5: We evaluate the performance of the optimizers in the real-world scenes. Our optimizer attains better quality in shorter training time.