Improving Gaussian Splatting with Localized Points Management

TL;DR

This work tackles limitations of Adaptive Density Control in 3D Gaussian Splatting, where non-uniform scene complexity and ill-conditioned points hinder accurate geometry. It introduces Localized Point Management (LPM), a plugin that uses image-rendering error maps and multiview geometry to identify error-contributing 3D zones, enabling localized point densification and opacity resets within those zones. By leveraging cross-view region correspondences via LightGlue and cone-based 2D-to-3D projection, LPM effectively calibrates geometry while pruning to control growth, achieving state-of-the-art rendering quality on static 3D and dynamic 4D datasets (e.g., Tanks & Temples, Neural 3D Video) with real-time performance. The method demonstrates strong quantitative gains in PSNR/SSIM/LPIPS and qualitative improvements in challenging regions (transparency, depth) across both 3DGS and SpaceTimeGS baselines, highlighting its practical impact as a versatile enhancement for Gaussian Splatting pipelines.

Abstract

Point management is critical for optimizing 3D Gaussian Splatting models, as point initiation (e.g., via structure from motion) is often distributionally inappropriate. Typically, Adaptive Density Control (ADC) algorithm is adopted, leveraging view-averaged gradient magnitude thresholding for point densification, opacity thresholding for pruning, and regular all-points opacity reset. We reveal that this strategy is limited in tackling intricate/special image regions (e.g., transparent) due to inability of identifying all 3D zones requiring point densification, and lacking an appropriate mechanism to handle ill-conditioned points with negative impacts (e.g., occlusion due to false high opacity). To address these limitations, we propose a Localized Point Management (LPM) strategy, capable of identifying those error-contributing zones in greatest need for both point addition and geometry calibration. Zone identification is achieved by leveraging the underlying multiview geometry constraints, subject to image rendering errors. We apply point densification in the identified zones and then reset the opacity of the points in front of these regions, creating a new opportunity to correct poorly conditioned points. Serving as a versatile plugin, LPM can be seamlessly integrated into existing static 3D and dynamic 4D Gaussian Splatting models with minimal additional cost. Experimental evaluations validate the efficacy of our LPM in boosting a variety of existing 3D/4D models both quantitatively and qualitatively. Notably, LPM improves both static 3DGS and dynamic SpaceTimeGS to achieve state-of-the-art rendering quality while retaining real-time speeds, excelling on challenging datasets such as Tanks & Temples and the Neural 3D Video dataset.

Figures (6)

• Figure 1: Visualization of points behavior. 3DGS produces ill-conditioned Gaussians (red box) that occlude other valid points, resulting in noticeably incorrect depth estimation. LPM handles these ill-conditioned points to reduce negative impacts and further calibrate the geometry.
• Figure 2: Overview of our Localized Point Management (LPM). (a) We start with an image rendering error map versus the current view (the ground-truth). Concurrently, matching points are identified between the current view and a referred view sampled as an adjacent view via off-the-shelf feature mapping. (b) Subsequently, cross-view region mapping is then employed to locate the correspondence region in the refereed view. (c) For each pair of corresponded regions, we cast the rays through them at their respective camera views in the cone shape, and consider their intersection as the error source zone. The final step involves identifying under-optimized or ill-conditioned points within these zones, where under-optimized/empty places are densified, and ill-conditioned points are reset.
• Figure 3: Qualitative evaluation of our LPM across diverse static datasets barron2022miphedman2018deep. Our LPM improves 2DGS huang20242d and 3DGS kerbl20233d on these challenging scenarios, e.g. (a) Light artifacts, (b) Completeness in the distance, (c) Depth structure and (d) Mesh details. See red patches for highlighted visual differences.
• Figure 4: Effect of key operations of LPM. We show that the point addition operation effectively captures the geometric details in the scene; The point reset operation based on the error map further calibrate the geometry.
• Figure 5: Qualitative evaluation on dynamic Neural 3D Video dataset li2022neural. LPM improves STGS li2023spacetime for both scenes Transparent (e.g., window) and Dynamic movements (e.g., dog's tongue).