GaussianPro: 3D Gaussian Splatting with Progressive Propagation

TL;DR

GaussianPro tackles the initialization bottleneck of 3D Gaussian Splatting by introducing progressive propagation that propagates accurate depth and normals from well-modeled regions to textureless areas using 2D depth/normal maps and patch matching. The method combines a hybrid geometric representation with a planar constraint, enforcing alignment between rendered Gaussians and propagated geometry to produce more compact, accurate Gaussians. Empirical results on Waymo and MipNeRF360 show state-of-the-art rendering quality, notably a +1.15 dB PSNR improvement on Waymo, while maintaining efficient real-time-style rendering. This approach enhances the practicality of neural rendering for large-scale, textureless scenes and offers robustness to varying numbers of input views, with potential extensions to dynamic scenes.

Abstract

The advent of 3D Gaussian Splatting (3DGS) has recently brought about a revolution in the field of neural rendering, facilitating high-quality renderings at real-time speed. However, 3DGS heavily depends on the initialized point cloud produced by Structure-from-Motion (SfM) techniques. When tackling with large-scale scenes that unavoidably contain texture-less surfaces, the SfM techniques always fail to produce enough points in these surfaces and cannot provide good initialization for 3DGS. As a result, 3DGS suffers from difficult optimization and low-quality renderings. In this paper, inspired by classical multi-view stereo (MVS) techniques, we propose GaussianPro, a novel method that applies a progressive propagation strategy to guide the densification of the 3D Gaussians. Compared to the simple split and clone strategies used in 3DGS, our method leverages the priors of the existing reconstructed geometries of the scene and patch matching techniques to produce new Gaussians with accurate positions and orientations. Experiments on both large-scale and small-scale scenes validate the effectiveness of our method, where our method significantly surpasses 3DGS on the Waymo dataset, exhibiting an improvement of 1.15dB in terms of PSNR.

Figures (7)

• Figure 1: The sparse SfM points and less-constrained densification strategies of 3DGS pose challenges in optimizing 3D Gaussians, particularly for textureless areas. 3DGS generates incorrect Gaussians (blue circle) to be over-fitted on the training images, leading to a noticeable performance drop in novel view rendering with erroneous geometries.
• Figure 2: Progressive Propagation of Gaussian. Firstly, we render the depth and normal maps from the 3D Gaussians. Then we iteratively perform propagation operations on the rendered depths and normals to generate new depth and normal values (denoted as propagated depth and propagated normal) via patch matching techniques. We filter out the unreliable propagated depths and normals using geometric consistency, yielding filtered depths and filtered normals. Finally, we identify the regions where their rendered depths and normals significantly deviate from the filtered ones, indicating that existing Gaussians may not inaccurately capture the geometry and therefore need more Gaussians. Pixels in these regions are projected into the 3D space to initialize new Gaussians using the filtered depth and normal.
• Figure 3: Patch matching. To select the best plane candidate for pixel $p$ during propagation, we perform homography transformation between $p$ and each plane candidate, thus yielding the possible corresponding pixels of the neighboring view. The plane candidate that exhibits the highest color consistency between $p$ and its possible paired pixel is chosen to be the solution. The chosen plane candidate is used to update the depth and normal of pixel $p$.
• Figure 4: Visual comparisons with 3DGS on novel view synthesis. The rendered image of 3DGS contains severe artifacts since the Gaussian spheres are out of order and do not accurately model the true geometry. On the contrary, our method faithfully captures the details of the road, and its Gaussian spheres are more compact and orderly.
• Figure 5: Rendering results on the Waymo (left) and MipNeRF360 (right) datasets. Compared to 3DGS, we have achieved a noticeable improvement in both texture-less surfaces and sharp details.