Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

S2D: Sparse to Dense Lifting for 3D Reconstruction with Minimal Inputs

Yuzhou Ji, Qijian Tian, He Zhu, Xiaoqi Jiang, Guangzhi Cao, Lizhuang Ma, Yuan Xie, Xin Tan

TL;DR

Sparse to Dense lifting (S2D), a novel pipeline that bridges the two representations and achieves high-quality 3DGS reconstruction with minimal inputs and enables minimal input requirements for 3DGS applications is introduced.

Abstract

Explicit 3D representations have already become an essential medium for 3D simulation and understanding. However, the most commonly used point cloud and 3D Gaussian Splatting (3DGS) each suffer from non-photorealistic rendering and significant degradation under sparse inputs. In this paper, we introduce Sparse to Dense lifting (S2D), a novel pipeline that bridges the two representations and achieves high-quality 3DGS reconstruction with minimal inputs. Specifically, the S2D lifting is two-fold. We first present an efficient one-step diffusion model that lifts sparse point cloud for high-fidelity image artifact fixing. Meanwhile, to reconstruct 3D consistent scenes, we also design a corresponding reconstruction strategy with random sample drop and weighted gradient for robust model fitting from sparse input views to dense novel views. Extensive experiments show that S2D achieves the best consistency in generating novel view guidance and first-tier sparse view reconstruction quality under different input sparsity. By reconstructing stable scenes with the least possible captures among existing methods, S2D enables minimal input requirements for 3DGS applications.

S2D: Sparse to Dense Lifting for 3D Reconstruction with Minimal Inputs

TL;DR

Sparse to Dense lifting (S2D), a novel pipeline that bridges the two representations and achieves high-quality 3DGS reconstruction with minimal inputs and enables minimal input requirements for 3DGS applications is introduced.

Abstract

Explicit 3D representations have already become an essential medium for 3D simulation and understanding. However, the most commonly used point cloud and 3D Gaussian Splatting (3DGS) each suffer from non-photorealistic rendering and significant degradation under sparse inputs. In this paper, we introduce Sparse to Dense lifting (S2D), a novel pipeline that bridges the two representations and achieves high-quality 3DGS reconstruction with minimal inputs. Specifically, the S2D lifting is two-fold. We first present an efficient one-step diffusion model that lifts sparse point cloud for high-fidelity image artifact fixing. Meanwhile, to reconstruct 3D consistent scenes, we also design a corresponding reconstruction strategy with random sample drop and weighted gradient for robust model fitting from sparse input views to dense novel views. Extensive experiments show that S2D achieves the best consistency in generating novel view guidance and first-tier sparse view reconstruction quality under different input sparsity. By reconstructing stable scenes with the least possible captures among existing methods, S2D enables minimal input requirements for 3DGS applications.
Paper Structure (26 sections, 15 equations, 10 figures, 4 tables, 1 algorithm)

This paper contains 26 sections, 15 equations, 10 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Gaussian Splatting
  4. Universal 3D reconstruction
  5. Reconstruction with diffusion prior
  6. Method
  7. Overview
  8. Initialization
  9. Novel view artifact fixer
  10. Reconstruction with novel guidance
  11. Experiments
  12. Implementation Details
  13. Datasets and Metrics
  14. Comparison on in-the-wild scenes
  15. Comparison on driving scenes
  16. ...and 11 more sections

Figures (10)

  • Figure 1: We demonstrate S2D in different situations. S2D supports reconstruction with inputs of different density. The 3DGS scene reconstructed through S2D is free of severe artifacts that occur in traditional reconstruction under sparse inputs. S2D also outperforms state-of-the-art scene enhancer (DIFIX) and direct video generation conditioned on camera poses (SEVA).
  • Figure 2: S2D reconstruction pipeline and model architecture of artifact fixer.
  • Figure 3: Evaluation on parameter $\alpha$ and $\beta$.
  • Figure 4: Qualitative results in different situations.
  • Figure 5: Comparison of artifact removal with ablations on guidance mixing.
  • ...and 5 more figures