Geometry and Perception Guided Gaussians for Multiview-consistent 3D Generation from a Single Image

TL;DR

This work tackles single-view 3D generation by integrating geometry and perception priors into 3D Gaussian Splatting to achieve multiview consistency without additional training. It introduces three Gaussian branches initialized from geometry, perception, and random noise, and enhances stability with a Stable SDS loss plus a reprojection-based depth-consistency loss. Perceptual augmentation and a reprojection strategy jointly improve multiview fidelity and geometric detail, validated on the Google Scanned Object dataset with strong novel-view synthesis and reconstruction results. The approach offers a training-efficient path to high-quality 3D assets from a single image, suitable for rapid generation and practical deployment.

Abstract

Generating realistic 3D objects from single-view images requires natural appearance, 3D consistency, and the ability to capture multiple plausible interpretations of unseen regions. Existing approaches often rely on fine-tuning pretrained 2D diffusion models or directly generating 3D information through fast network inference or 3D Gaussian Splatting, but their results generally suffer from poor multiview consistency and lack geometric detail. To tackle these issues, we present a novel method that seamlessly integrates geometry and perception information without requiring additional model training to reconstruct detailed 3D objects from a single image. Specifically, we incorporate geometry and perception priors to initialize the Gaussian branches and guide their parameter optimization. The geometry prior captures the rough 3D shapes, while the perception prior utilizes the 2D pretrained diffusion model to enhance multiview information. Subsequently, we introduce a stable Score Distillation Sampling for fine-grained prior distillation to ensure effective knowledge transfer. The model is further enhanced by a reprojection-based strategy that enforces depth consistency. Experimental results show that we outperform existing methods on novel view synthesis and 3D reconstruction, demonstrating robust and consistent 3D object generation.

Figures (7)

• Figure 1: Our method is able to generate multiview-consistent 3D objects from a single-view input image. For each input image, our method generates point clouds, meshes and multiview images from any views without additional training.
• Figure 2: The architecture of the proposed method. We use the geometry prior, perception prior and Gaussian noise to initialize three different Gaussian branches. At each training step, each Gaussian branch is supervised by the input image and augmented images generated by the perception prior.
• Figure 3: Conceptual figure of the proposed stable SDS between two Gaussian branches. The target distribution in the figure represents the conditional distribution of Gaussian parameters relative to the input image. Let $\delta_{\text{ideal}}=\delta_{\text{real}}+\delta_{\text{cond}}$ denote the ideal gradient update direction, while $\delta_{\text{real}}=\delta_{\text{ideal}}+\delta_{\text{bias}}$ represents the actual update direction obtained in practice. Compared to $\nabla_{\theta}\mathcal{L}_{\text{SDS}}$, $\nabla_{\theta}\mathcal{L}_{\text{S-SDS}}$ further constrains the stochasticity of bias perturbations, thereby promoting more stable parameter updates that converge toward higher-density regions of the target distribution.
• Figure 4: The process of calculating the reprojection loss. For a Gaussian branch, we sample a pair of views $\mathbf{p}_1$ and $\mathbf{p}_2$ with minimal viewpoint difference. We render images from the two views and calculate their pixel-level correspondence. Finally, we calculate the reprojection loss according to the correspondence.
• Figure 5: Qualitative comparison with MVD-Fusion and DreamGaussian on synthesized multiview images.