ResGS: Residual Densification of 3D Gaussian for Efficient Detail Recovery

TL;DR

ResGS identifies inflexibilities in 3D-GS densification (split/clone) and introduces a residual split densification in a coarse-to-fine pipeline. The method couples residual-based adaptive detail refinement with image-pyramid supervision and a progressively shifting densification focus to decouple geometry coverage from detail recovery. Empirical results demonstrate state-of-the-art rendering quality on Mip-NeRF360 and strong performance on Tanks&Temples and Deep Blending, with consistent gains when applying residual split across variants. The work offers a versatile, broadly compatible approach for improving 3D-GS-based novel view synthesis with potential for wider adoption in related 3D-GS applications.

Abstract

Recently, 3D Gaussian Splatting (3D-GS) has prevailed in novel view synthesis, achieving high fidelity and efficiency. However, it often struggles to capture rich details and complete geometry. Our analysis reveals that the 3D-GS densification operation lacks adaptiveness and faces a dilemma between geometry coverage and detail recovery. To address this, we introduce a novel densification operation, residual split, which adds a downscaled Gaussian as a residual. Our approach is capable of adaptively retrieving details and complementing missing geometry. To further support this method, we propose a pipeline named ResGS. Specifically, we integrate a Gaussian image pyramid for progressive supervision and implement a selection scheme that prioritizes the densification of coarse Gaussians over time. Extensive experiments demonstrate that our method achieves SOTA rendering quality. Consistent performance improvements can be achieved by applying our residual split on various 3D-GS variants, underscoring its versatility and potential for broader application in 3D-GS-based applications.

Figures (7)

• Figure 1: We propose ResGS, a pipeline to boost the rendering quality of 3D-GS Kerbl20233DGS while improving efficiency. (a) The 3D-GS densification operations, split and clone, rely on binary selection and cannot adaptively address the challenges they encounter. (b) Our ResGS employs a single densification operation, residual split, that adaptively addresses under-reconstruction and over-reconstruction. (c) 3D-GS tends to generate Gaussians with sizes in small variation and low rendering quality. (d) From the same initialization, our method can capture fine details and retrieve sufficient geometry effectively while reducing redundancy.
• Figure 2: An illustration of the limitation of split and clone.(a) selects between split and clone based on a high threshold, (b) a low threshold, and in (c) the densification operation is replaced as residual split. We provide visualizations of the final rendered image. A high threshold ensures sufficient structural coverage but causes over-reconstruction, leading to blurry rendering. Conversely, a low threshold mitigates blurry rendering but results in under-reconstruction, compromising coverage and creating a difficult trade-off. Our method, however, adaptively tackles both issues without facing this trade-off.
• Figure 3: Overview of our ResGS pipeline. (a) The core of our pipeline, residual split, involves adding a downscaled replicate and then reducing the opacity of the original Gaussian. (b) We assign initial Gaussians a temporary attribute $l_i=0$ for densification selection, which is discarded after training. Next, the pipeline is split into $L$ stages, with each stage trained on images downscaled using an image pyramid. Each single stage is further divided evenly into $K$ substages, for selecting Gaussians to densify. (c) The points selected for densification are determined by the substage $k$, $l_i$, and viewspace gradients of Gaussians. Residual split is performed upon selected Gaussians, and $l_j$ would be assigned to new Gaussians.
• Figure 4: Intermediate results after applying residual split. We saved the Gaussians at 3,000 and 6,000 iterations after applying residual split. As observed, residual split can adaptively handle over-reconstruction and under-reconstruction by refining blurry regions with finer Gaussians while also effectively reconstructing missing structures.
• Figure 5: Qualitative comparisons of ResGS with three 3D-GS variations on a variety of indoor and outdoor scenes. Our approach captures more intrinsic details and acquires more complete geometry in complex scenes.