HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction

TL;DR

This paper proposes a novel framework, HiSplat, which introduces a hierarchical manner in generalizable 3D Gaussian Splatting to construct hierarchical 3D Gaussians via a coarse-to-fine strategy, and significantly enhances reconstruction quality and cross-dataset generalization compared to prior single-scale methods.

Abstract

Reconstructing 3D scenes from multiple viewpoints is a fundamental task in stereo vision. Recently, advances in generalizable 3D Gaussian Splatting have enabled high-quality novel view synthesis for unseen scenes from sparse input views by feed-forward predicting per-pixel Gaussian parameters without extra optimization. However, existing methods typically generate single-scale 3D Gaussians, which lack representation of both large-scale structure and texture details, resulting in mislocation and artefacts. In this paper, we propose a novel framework, HiSplat, which introduces a hierarchical manner in generalizable 3D Gaussian Splatting to construct hierarchical 3D Gaussians via a coarse-to-fine strategy. Specifically, HiSplat generates large coarse-grained Gaussians to capture large-scale structures, followed by fine-grained Gaussians to enhance delicate texture details. To promote inter-scale interactions, we propose an Error Aware Module for Gaussian compensation and a Modulating Fusion Module for Gaussian repair. Our method achieves joint optimization of hierarchical representations, allowing for novel view synthesis using only two-view reference images. Comprehensive experiments on various datasets demonstrate that HiSplat significantly enhances reconstruction quality and cross-dataset generalization compared to prior single-scale methods. The corresponding ablation study and analysis of different-scale 3D Gaussians reveal the mechanism behind the effectiveness. Project website: https://open3dvlab.github.io/HiSplat/

Figures (9)

• Figure 1: Comparison between HiSplat and previous methods. HiSplat constructs hierarchical 3D Gaussians which can better represent large-scale structures (more accurate location and less crack), and texture details (fewer artefacts and less blurriness).
• Figure 2: The overall framework of HiSplat. For simplicity, the situation with two input images is illustrated. HiSplat utilizes a shared U-Net backbone to extract different-scale features. With these features, three processing stages predict pixel-aligned Gaussian parameters with different scales, respectively. Error aware module and modulating fusion module perceive the errors in the early stages and guide the Gaussians in the later stages for compensation and repair. Finally, the fusing hierarchical Gaussians can reconstruct both the large-scale structure and texture details.
• Figure 3: Qualitative comparison of generalization ability. For the scenes out of training distribution, HiSplat can generate higher-quality novel-view images. More comparison is provided in \ref{['sec_app:more_com']}.
• Figure 4: Comparison of Gaussian primitives in different stages on DTU. HisPlat can gradually generate large-scale solid Gaussians as "bone" and small-scale transplant Gaussians as "flesh", confirming better rendering quality and geometry.
• Figure 5: Comparison of rendering images from different stages on RealEstate10K. HiSplat can perceive the error, and utilize Gaussians in the later stages to add details and correct errors gradually.