Consistency Diffusion Models for Single-Image 3D Reconstruction with Priors

TL;DR

This work targets single-image 3D point cloud reconstruction by fusing 3D priors and 2D priors within a diffusion framework. The Consistency Diffusion Model (CDM) introduces a 3D Priors Constraint that tightens ELBO by leveraging multi-view depth priors, and integrates 2D priors derived from the input image to guide training, avoiding drift between training and sampling. Extensive experiments on ShapeNet and Co3D demonstrate state-of-the-art performance and illustrate the complementary benefits of 2D and 3D priors, with depth and contour cues playing dataset-dependent roles. The approach achieves improved reconstruction fidelity while maintaining reasonable training overhead, and points to future work in text-guided point cloud editing.

Abstract

This paper delves into the study of 3D point cloud reconstruction from a single image. Our objective is to develop the Consistency Diffusion Model, exploring synergistic 2D and 3D priors in the Bayesian framework to ensure superior consistency in the reconstruction process, a challenging yet critical requirement in this field. Specifically, we introduce a pioneering training framework under diffusion models that brings two key innovations. First, we convert 3D structural priors derived from the initial 3D point cloud as a bound term to increase evidence in the variational Bayesian framework, leveraging these robust intrinsic priors to tightly govern the diffusion training process and bolster consistency in reconstruction. Second, we extract and incorporate 2D priors from the single input image, projecting them onto the 3D point cloud to enrich the guidance for diffusion training. Our framework not only sidesteps potential model learning shifts that may arise from directly imposing additional constraints during training but also precisely transposes the 2D priors into the 3D domain. Extensive experimental evaluations reveal that our approach sets new benchmarks in both synthetic and real-world datasets. The code is included with the submission.

Figures (8)

• Figure 1: Illustration of reconstruction results comparison and network structures. Left: the reconstruction results of $\text{PC}^2$, BDM, and our CDM. Right: the network structures of the three approaches. BDM focuses on randomly merging the outputs of two models during the sampling phase, while our method leverages tailored 2D and 3D priors to promote structural consistency on the reconstruction process during training.
• Figure 2: Model structure of conventional diffusion model and our consistency diffusion model.
• Figure 3: Illustration of rendered "teddybear" image from 4 different viewpoints. Top: rendered point cloud images. Bottom: rendered point cloud depth images.
• Figure 4: Illustrative structure for incorporating 2D and 3D priors. The 2D priors (upper part) are concatenated with the image features and mapped onto the point cloud as conditions. The 3D priors (lower part) are transformed into depth images of the point cloud at time steps $x_{0}$ and $x_{t}$. The bound term is transformed into the distances between corresponding depth images for increasing the ELBO.
• Figure 5: Visual comparison on the ShapeNet dataset. The first column displays the input image. We compare the reconstructed point clouds from two different viewpoints. Intuitively, $\text{PC}^2$ produces ambiguous results due to weak constraints, and BDM, which introduces class-level priors, fails to effectively control reconstruction consistency.