Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frequency-Adaptive Sharpness Regularization for Improving 3D Gaussian Splatting Generalization

Youngsik Yun, Dongjun Gu, Youngjung Uh

TL;DR

This work tackles the generalization gap of 3D Gaussian Splatting in sparse-view novel views by reframing optimization as a generalization problem and introducing Frequency-Adaptive Sharpness Regularization (FASR). FASR redefines sharpness regularization to be frequency-aware, applying per-Gaussian perturbations and weighting guided by a local frequency map, thereby preserving high-frequency detail while improving generalization. Across LLFF and MipNeRF-360 benchmarks and multiple baselines, FASR yields consistent improvements in PSNR, SSIM, LPIPS, and AVGE, and ablation analyses highlight the importance of per-Gaussian treatment and frequency-aware perturbations. The method remains complementary to existing priors and is extensible to NeRF-based models and temporally sparse, dynamic scenes, offering practical gains for sparse-view 3D reconstruction and beyond.

Abstract

Despite 3D Gaussian Splatting (3DGS) excelling in most configurations, it lacks generalization across novel viewpoints in a few-shot scenario because it overfits to the sparse observations. We revisit 3DGS optimization from a machine learning perspective, framing novel view synthesis as a generalization problem to unseen viewpoints-an underexplored direction. We propose Frequency-Adaptive Sharpness Regularization (FASR), which reformulates the 3DGS training objective, thereby guiding 3DGS to converge toward a better generalization solution. Although Sharpness-Aware Minimization (SAM) similarly reduces the sharpness of the loss landscape to improve generalization of classification models, directly employing it to 3DGS is suboptimal due to the discrepancy between the tasks. Specifically, it hinders reconstructing high-frequency details due to excessive regularization, while reducing its strength leads to under-penalizing sharpness. To address this, we reflect the local frequency of images to set the regularization weight and the neighborhood radius when estimating the local sharpness. It prevents floater artifacts in novel viewpoints and reconstructs fine details that SAM tends to oversmooth. Across datasets with various configurations, our method consistently improves a wide range of baselines. Code will be available at https://bbangsik13.github.io/FASR.

Frequency-Adaptive Sharpness Regularization for Improving 3D Gaussian Splatting Generalization

TL;DR

This work tackles the generalization gap of 3D Gaussian Splatting in sparse-view novel views by reframing optimization as a generalization problem and introducing Frequency-Adaptive Sharpness Regularization (FASR). FASR redefines sharpness regularization to be frequency-aware, applying per-Gaussian perturbations and weighting guided by a local frequency map, thereby preserving high-frequency detail while improving generalization. Across LLFF and MipNeRF-360 benchmarks and multiple baselines, FASR yields consistent improvements in PSNR, SSIM, LPIPS, and AVGE, and ablation analyses highlight the importance of per-Gaussian treatment and frequency-aware perturbations. The method remains complementary to existing priors and is extensible to NeRF-based models and temporally sparse, dynamic scenes, offering practical gains for sparse-view 3D reconstruction and beyond.

Abstract

Despite 3D Gaussian Splatting (3DGS) excelling in most configurations, it lacks generalization across novel viewpoints in a few-shot scenario because it overfits to the sparse observations. We revisit 3DGS optimization from a machine learning perspective, framing novel view synthesis as a generalization problem to unseen viewpoints-an underexplored direction. We propose Frequency-Adaptive Sharpness Regularization (FASR), which reformulates the 3DGS training objective, thereby guiding 3DGS to converge toward a better generalization solution. Although Sharpness-Aware Minimization (SAM) similarly reduces the sharpness of the loss landscape to improve generalization of classification models, directly employing it to 3DGS is suboptimal due to the discrepancy between the tasks. Specifically, it hinders reconstructing high-frequency details due to excessive regularization, while reducing its strength leads to under-penalizing sharpness. To address this, we reflect the local frequency of images to set the regularization weight and the neighborhood radius when estimating the local sharpness. It prevents floater artifacts in novel viewpoints and reconstructs fine details that SAM tends to oversmooth. Across datasets with various configurations, our method consistently improves a wide range of baselines. Code will be available at https://bbangsik13.github.io/FASR.

Paper Structure

This paper contains 34 sections, 1 theorem, 8 equations, 10 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Sparse-view reconstruction
  4. Flatness and generalization
  5. Reconstructing with perturbation
  6. Method
  7. Preliminary: Sharpness-Aware Minimization
  8. Applying SAM to 3DGS
  9. Key hypothesis.
  10. Frequency-Adaptive Sharpness Regularization
  11. Separate sharpness per-Gaussian
  12. Frequency-adaptive perturbation magnitude
  13. Frequency-adaptive sharpness weighting
  14. Experiments
  15. Dataset.
  16. ...and 19 more sections

Key Result

Theorem 1

For any $\rho>0$, with training set $\mathcal{S}$ from data distribution $\mathscr{D}$, where $h: \mathbb{R}_{+}\rightarrow \mathbb{R}_{+}$ is a strictly increasing function (under some technical conditions on $L_\mathscr{D}(\boldsymbol{w})$).

Figures (10)

  • Figure 1: Overview. Our proposed optimization algorithm improves generalization. Given eight training views rendered from the lego scene in Blender synthetic dataset midenhall2020nerf, our method maintains low Average Error niemeyer2022regnerf across interpolated novel views, whereas 3DGS exhibits overfitting. Plots are means and standard deviations over ten runs.
  • Figure 2: Conceptual 1D Loss Landscape of Flat and Sharp Minima. Flat minimum better generalize then sharp minimum.
  • Figure 3: Overview of our proposed method.
  • Figure 4: Qualitative comparison. Please zoom on the insets in red boxes to compare reconstruction quality.
  • Figure 5: Ablation study. FAP and FAS denote frequency-adaptive perturbation magnitude and frequency-adaptive sharpness weighting, respectively. "3DGS", "SAM", and "w/o FAS & FAP" produce inaccurate geometry (red box). All except "Ours Full" show blurry results (yellow box).
  • ...and 5 more figures

Theorems & Definitions (1)

  • Theorem 1