HeroGS: Hierarchical Guidance for Robust 3D Gaussian Splatting under Sparse Views

TL;DR

HeroGS, Hierarchical Guidance for Robust 3D Gaussian Splatting is proposed, a unified framework that establishes hierarchical guidance across the image, feature, and parameter levels and effectively constrains and optimizes the overall Gaussian distributions, thereby enhancing both structural fidelity and rendering quality.

Abstract

3D Gaussian Splatting (3DGS) has recently emerged as a promising approach in novel view synthesis, combining photorealistic rendering with real-time efficiency. However, its success heavily relies on dense camera coverage; under sparse-view conditions, insufficient supervision leads to irregular Gaussian distributions, characterized by globally sparse coverage, blurred background, and distorted high-frequency areas. To address this, we propose HeroGS, Hierarchical Guidance for Robust 3D Gaussian Splatting, a unified framework that establishes hierarchical guidance across the image, feature, and parameter levels. At the image level, sparse supervision is converted into pseudo-dense guidance, globally regularizing the Gaussian distributions and forming a consistent foundation for subsequent optimization. Building upon this, Feature-Adaptive Densification and Pruning (FADP) at the feature level leverages low-level features to refine high-frequency details and adaptively densifies Gaussians in background regions. The optimized distributions then support Co-Pruned Geometry Consistency (CPG) at parameter level, which guides geometric consistency through parameter freezing and co-pruning, effectively removing inconsistent splats. The hierarchical guidance strategy effectively constrains and optimizes the overall Gaussian distributions, thereby enhancing both structural fidelity and rendering quality. Extensive experiments demonstrate that HeroGS achieves high-fidelity reconstructions and consistently surpasses state-of-the-art baselines under sparse-view conditions.

Figures (9)

• Figure 1: Motivation of HeroGS. (Left) Image-level pseudo-labels bridge the gap between sparse and dense supervision, yielding a more complete Gaussian distributions. (Right) FADP and CPG refine inaccurate Gaussians by enhancing distributions closer to ground-truth geometry and pruning inconsistencies.
• Figure 2: HeroGS Overview. Initialized from SfM, our framework operates across three levels. (1) Image-level Guidance: A set of intermediate RGB frames is synthesized from sparse inputs, offering pseudo-dense guidance that globally regularizes the Gaussian distributions and, at the feature level, manifests as patch-based Gaussian numbers $\mathcal{C}$. (2) Feature-level Refinement: The Feature-Adaptive Densification and Pruning (FADP) leverages edge- and patch-aware features from training views to enhance high-frequency and background regions, while suppressing redundant Gaussians and dilivering finer details for next level. (3) Parameter-level Consistency: The Co-Pruned Geometry Consistency (CPG) employs auxiliary Gaussian fields with partially frozen parameters to perform co-pruning, eliminating geometrically inconsistent splats. These levels form a hierarchical guidance with interconnections (dashed lines) that jointly constrains and optimizes the Gaussian field for improved structural fidelity.
• Figure 3: Removing inconsistent Gaussians. Columns 3 and 4: original training views and frame-interpolated pseudo-labels, respectively. Although pseudo-labels provide overall supervisory signals, they may lack accuracy in fine details and fail to offer precise guidance for high-frequency geometric structures. Columns 1 and 2: novel-view renderings produced by full pipeline versus those without CPG. It suppresses drifts, yielding sharper edges, coherent surfaces and markedly improved spatial fidelity.
• Figure 4: Qualitative Comparison on LLFF (3 training views). Under extreme view sparsity, 3DGS kerbl20233d collapses into severe artifacts and blurred geometry. DRGS and FSGS recover coarse structure yet still exhibit over-smoothed textures and noisy backgrounds. In contrast, HeroGS guides the model from complementary levels to deliver markedly sharper object boundaries, richer high-frequency textures, and distinctly clearer distant regions—demonstrating the efficacy of our full pipeline.
• Figure 5: Comparison with and without image and feature level guidance. Columns 1 and 2: novel-view renderings produced by baseline versus baseline with pseudo-labels and FADP. Columns 3 and 4: their corresponding Gaussians.