PSRGS:Progressive Spectral Residual of 3D Gaussian for High-Frequency Recovery

TL;DR

PSRGS addresses the over-reconstruction artifacts of 3D Gaussian Splatting on large-scale remote sensing data by decoupling the scene into low-frequency geometry and high-frequency texture using a spectral residual saliency map $H(x,y;\mathbf{S})$, learned with an MLP. It applies a depth-aware geometry constraint and a gradient-guided high-frequency texture recovery, complemented by a multi-view perceptual loss from a pre-trained network to sharpen texture while preserving geometry, with a rasterization model $C = \sum_i T_i \sigma_i c_i$. The approach yields improved high-frequency detail and geometric fidelity, demonstrated on mip-NeRF 360 and Lund datasets, and outperforms prior 3D GS variants in large scenes. The method offers practical gains for large-scale remote sensing and other complex 3D scene applications.

Abstract

3D Gaussian Splatting (3D GS) achieves impressive results in novel view synthesis for small, single-object scenes through Gaussian ellipsoid initialization and adaptive density control. However, when applied to large-scale remote sensing scenes, 3D GS faces challenges: the point clouds generated by Structure-from-Motion (SfM) are often sparse, and the inherent smoothing behavior of 3D GS leads to over-reconstruction in high-frequency regions, where have detailed textures and color variations. This results in the generation of large, opaque Gaussian ellipsoids that cause gradient artifacts. Moreover, the simultaneous optimization of both geometry and texture may lead to densification of Gaussian ellipsoids at incorrect geometric locations, resulting in artifacts in other views. To address these issues, we propose PSRGS, a progressive optimization scheme based on spectral residual maps. Specifically, we create a spectral residual significance map to separate low-frequency and high-frequency regions. In the low-frequency region, we apply depth-aware and depth-smooth losses to initialize the scene geometry with low threshold. For the high-frequency region, we use gradient features with higher threshold to split and clone ellipsoids, refining the scene. The sampling rate is determined by feature responses and gradient loss. Finally, we introduce a pre-trained network that jointly computes perceptual loss from multiple views, ensuring accurate restoration of high-frequency details in both Gaussian ellipsoids geometry and color. We conduct experiments on multiple datasets to assess the effectiveness of our method, which demonstrates competitive rendering quality, especially in recovering texture details in high-frequency regions.

Figures (4)

• Figure 1: The over-reconstruction tendency of 3D GS tends to generate large Gaussian ellipsoids, leading to artifacts in the scene. In contrast, our method manages the geometric structure effectively, reconstructing the scene more accurately and eliminating artifacts
• Figure 2: Overview of Our PRSGS Pipeline. (a) Spectral Salient Residual Decoupling. We perform a spectral transform, where the smoothed residual of the logarithmic magnitude spectrum is combined with the original phase spectrum to generate the salient map. The standard deviation for filtering is learned by an MLP network. (b) Low-Frequency Processing Stage. A higher threshold is applied for densification, and depth perception loss and smoothness loss are used to constrain the scene's depth and normal maps. (c) High-Frequency Processing Stage. Gradient feature maps of the image are generated, and densification is performed with a lower threshold along the gradient direction. Additionally, a pre-trained network is used to compute the multi-view geometric perception loss, incorporating plane constraints to reduce the generation of erroneous Gaussian points.
• Figure 3: Our Progressive Rendering Scheme. (a) We render the high-frequency residual regions separately, while the surrounding geometric structure is not included (yellow ellipse). At this stage, some of the geometric structure is also problematic (red box). (b) The final result of the progressive rendering scheme. Not only have we accurately recovered the entire scene's geometric structure, but we have also effectively eliminated the artifacts present during separate rendering.
• Figure 4: Visual comparisons with competing methods, showing that our approach better removes artifacts