S2Gaussian: Sparse-View Super-Resolution 3D Gaussian Splatting

TL;DR

This work tackles recovering high-fidelity 3D scenes from highly constrained inputs by introducing S2Gaussian, a two-stage framework that first builds a low-resolution Gaussian Splatting representation from sparse views and depth, then densifies to high-resolution Gaussians via Gaussian Shuffle Split. The HR Gaussians are refined using super-resolved imagery from both original views and pseudo-views, with a blur-free inconsistency modeling module and a 3D robust optimization strategy to mitigate multi-view inconsistencies and erroneous updates. Key contributions include the Gaussian Shuffle Split for compact, detail-rich initialization, a dedicated inconsistency-aware refinement pipeline, and robust optimization that yields superior geometry and texture quality, achieving state-of-the-art results on Blender, LLFF, and Mip-NeRF 360 under 4x sparse-view super-resolution. This framework advances practical 3D reconstruction in scenarios where both viewpoint sparsity and image blur limit performance, enabling more consistent rendering and finer details in real-world applications.

Abstract

In this paper, we aim ambitiously for a realistic yet challenging problem, namely, how to reconstruct high-quality 3D scenes from sparse low-resolution views that simultaneously suffer from deficient perspectives and clarity. Whereas existing methods only deal with either sparse views or low-resolution observations, they fail to handle such hybrid and complicated scenarios. To this end, we propose a novel Sparse-view Super-resolution 3D Gaussian Splatting framework, dubbed S2Gaussian, that can reconstruct structure-accurate and detail-faithful 3D scenes with only sparse and low-resolution views. The S2Gaussian operates in a two-stage fashion. In the first stage, we initially optimize a low-resolution Gaussian representation with depth regularization and densify it to initialize the high-resolution Gaussians through a tailored Gaussian Shuffle Split operation. In the second stage, we refine the high-resolution Gaussians with the super-resolved images generated from both original sparse views and pseudo-views rendered by the low-resolution Gaussians. In which a customized blur-free inconsistency modeling scheme and a 3D robust optimization strategy are elaborately designed to mitigate multi-view inconsistency and eliminate erroneous updates caused by imperfect supervision. Extensive experiments demonstrate superior results and in particular establishing new state-of-the-art performances with more consistent geometry and finer details.

Figures (9)

• Figure 1: We present S2Gaussian, a novel framework capable of reconstructing high-quality 3D scenes for immersive rendering with only sparse and low-resolution input views. S2Gaussian demonstrates superior performance and yields high-fidelity and high-resolution scene reconstruction with sharp geometric and detailed textures, thus enjoying better functionality and practicality in realistic applications.
• Figure 2: Overview of S2Gaussian. The S2Gaussian initially optimizes an LR GS and densifies it to initialize the HR GS through a tailored Gaussian Shuffle Split operation. Then, the original sparse views along with the pseudo views rendered by the LR GS are super-resolved together to refine the high-resolution texture with 3D robust optimization. In which an inconsistency modeling module (IM) and a blur proposal module are incorporated to mitigate inconsistency and blurriness, aiming to create 3D scenes with high-fidelity texture details.
• Figure 3: Illustration of Gaussian Shuffle Split that utilizes a combination of six compact Gaussians to replace the original large Gaussian primitive, which can densify the 3D scene virtually without damaging the original 3D representation.
• Figure 4: Top: Visualization of the gradient variation under different quality views supervision and our proposed 3D robust optimization. Bottom: Illustration of the proposed 3D robust optimization strategy that takes the gradient of scaling $s$ as an example.
• Figure 5: Qualitative comparisons on Blender $\times$4 datasets with 8 input views. Our method produces more visually appealing results, successfully reconstructing intricate thin structures with fine-grained details. Best viewed at screen!