Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3DGS$^2$-TR: Scalable Second-Order Trust-Region Method for 3D Gaussian Splatting

Roger Hsiao, Yuchen Fang, Xiangru Huang, Ruilong Li, Hesam Rabeti, Zan Gojcic, Javad Lavaei, James Demmel, Sophia Shao

TL;DR

3DGS$^2$-TR is able to achieve better reconstruction quality on standard datasets, using 50% fewer training iterations compared to ADAM, while incurring less than 1GB of peak GPU memory overhead, enabling scalability to very large scenes and potentially to distributed training settings.

Abstract

We propose 3DGS$^2$-TR,a second-order optimizer for accelerating the scene training problem in 3D Gaussian Splatting (3DGS). Unlike existing second-order approaches that rely on explicit or dense curvature representations, such as 3DGS-LM (Höllein et al., 2025) or 3DGS2 (Lan et al., 2025), our method approximates curvature using only the diagonal of the Hessian matrix, efficiently via Hutchinson's method. Our approach is fully matrix-free and has the same complexity as ADAM (Kingma, 2024), $O(n)$ in both computation and memory costs. To ensure stable optimization in the presence of strong nonlinearity in the 3DGS rasterization process, we introduce a parameter-wise trust-region technique based on the squared Hellinger distance, regularizing updates to Gaussian parameters. Under identical parameter initialization and without densification, 3DGS$^2$-TR is able to achieve better reconstruction quality on standard datasets, using 50% fewer training iterations compared to ADAM, while incurring less than 1GB of peak GPU memory overhead (17% more than ADAM and 85% less than 3DGS-LM), enabling scalability to very large scenes and potentially to distributed training settings.

3DGS$^2$-TR: Scalable Second-Order Trust-Region Method for 3D Gaussian Splatting

TL;DR

3DGS-TR is able to achieve better reconstruction quality on standard datasets, using 50% fewer training iterations compared to ADAM, while incurring less than 1GB of peak GPU memory overhead, enabling scalability to very large scenes and potentially to distributed training settings.

Abstract

We propose 3DGS-TR,a second-order optimizer for accelerating the scene training problem in 3D Gaussian Splatting (3DGS). Unlike existing second-order approaches that rely on explicit or dense curvature representations, such as 3DGS-LM (Höllein et al., 2025) or 3DGS2 (Lan et al., 2025), our method approximates curvature using only the diagonal of the Hessian matrix, efficiently via Hutchinson's method. Our approach is fully matrix-free and has the same complexity as ADAM (Kingma, 2024), in both computation and memory costs. To ensure stable optimization in the presence of strong nonlinearity in the 3DGS rasterization process, we introduce a parameter-wise trust-region technique based on the squared Hellinger distance, regularizing updates to Gaussian parameters. Under identical parameter initialization and without densification, 3DGS-TR is able to achieve better reconstruction quality on standard datasets, using 50% fewer training iterations compared to ADAM, while incurring less than 1GB of peak GPU memory overhead (17% more than ADAM and 85% less than 3DGS-LM), enabling scalability to very large scenes and potentially to distributed training settings.
Paper Structure (29 sections, 49 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 29 sections, 49 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Novel View Synthesis
  4. Accelerating 3DGS Training and Rendering
  5. Second-Order Optimizers
  6. Background and Notations
  7. Review of Gaussian Splatting
  8. Optimization Problem Formulation
  9. Methodology
  10. Algorithm Design
  11. Trust-Region Based on Squared Hellinger Distance
  12. Squared Hellinger distance.
  13. Scale normalization.
  14. Parameter-wise trust-region radius.
  15. Bound on the mean $\mu=(\mu_x,\mu_y,\mu_z)$.
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Overview of our proposed method.
  • Figure 2: Example scene with a single elongated Gaussian splat. The $x$-axis and $y$-axis lie on the page; the $z$-axis comes out of the page. The green arrows mark the directions in which the Gaussian has more freedom; while the red arrows denote otherwise.
  • Figure 3: A single Gaussian fitting example where the Gaussian splat is initialized with a small perturbation. Images from left to right show the progression of optimization to fit the perturbed splat to the ground truth splat (denoted with the orange circle). Both ADAM and $\text{3DGS}^2$-TR eventually recover the Gaussian parameters close to the ground truth, but $\text{3DGS}^2$-TR deforms the Gaussian much less due to the trust-region bound, which increases the stability of training.
  • Figure 4: Qualitative comparison of different methods of the truck, playroom, and room scenes. The red boxes highlight regions where $\text{3DGS}^2$-TR significantly outperforms other methods.