Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior

Zike Wu, Pan Zhou, Xuanyu Yi, Xiaoding Yuan, Hanwang Zhang

TL;DR

Consistent3D addresses the instability of SDS-based text-to-3D generation by replacing stochastic SDE sampling with deterministic ODE-based sampling. It introduces Consistency Distillation Sampling (CDS), a loss that distills a deterministic ODE flow into the 3D model by comparing two adjacent samples along the ODE trajectory under a fixed perturbation, yielding a $\mathcal{O}(\Delta)$ error bound and more stable guidance. The method employs a coarse-to-fine pipeline (NeRF-based representation at the low-resolution stage and mesh refinement at high resolution) and demonstrates superior fidelity and view-consistency on open prompts and 360° scenes, outperforming DreamFusion, Magic3D, and ProlificDreamer in CLIP R-precision and qualitative quality. The work provides both theoretical justification and practical algorithms for reliable, high-fidelity text-to-3D generation with deterministic priors, enabling faster generation and diversification across object and scene scales.

Abstract

Score distillation sampling (SDS) and its variants have greatly boosted the development of text-to-3D generation, but are vulnerable to geometry collapse and poor textures yet. To solve this issue, we first deeply analyze the SDS and find that its distillation sampling process indeed corresponds to the trajectory sampling of a stochastic differential equation (SDE): SDS samples along an SDE trajectory to yield a less noisy sample which then serves as a guidance to optimize a 3D model. However, the randomness in SDE sampling often leads to a diverse and unpredictable sample which is not always less noisy, and thus is not a consistently correct guidance, explaining the vulnerability of SDS. Since for any SDE, there always exists an ordinary differential equation (ODE) whose trajectory sampling can deterministically and consistently converge to the desired target point as the SDE, we propose a novel and effective "Consistent3D" method that explores the ODE deterministic sampling prior for text-to-3D generation. Specifically, at each training iteration, given a rendered image by a 3D model, we first estimate its desired 3D score function by a pre-trained 2D diffusion model, and build an ODE for trajectory sampling. Next, we design a consistency distillation sampling loss which samples along the ODE trajectory to generate two adjacent samples and uses the less noisy sample to guide another more noisy one for distilling the deterministic prior into the 3D model. Experimental results show the efficacy of our Consistent3D in generating high-fidelity and diverse 3D objects and large-scale scenes, as shown in Fig. 1. The codes are available at https://github.com/sail-sg/Consistent3D.

Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior

TL;DR

Consistent3D addresses the instability of SDS-based text-to-3D generation by replacing stochastic SDE sampling with deterministic ODE-based sampling. It introduces Consistency Distillation Sampling (CDS), a loss that distills a deterministic ODE flow into the 3D model by comparing two adjacent samples along the ODE trajectory under a fixed perturbation, yielding a error bound and more stable guidance. The method employs a coarse-to-fine pipeline (NeRF-based representation at the low-resolution stage and mesh refinement at high resolution) and demonstrates superior fidelity and view-consistency on open prompts and 360° scenes, outperforming DreamFusion, Magic3D, and ProlificDreamer in CLIP R-precision and qualitative quality. The work provides both theoretical justification and practical algorithms for reliable, high-fidelity text-to-3D generation with deterministic priors, enabling faster generation and diversification across object and scene scales.

Abstract

Score distillation sampling (SDS) and its variants have greatly boosted the development of text-to-3D generation, but are vulnerable to geometry collapse and poor textures yet. To solve this issue, we first deeply analyze the SDS and find that its distillation sampling process indeed corresponds to the trajectory sampling of a stochastic differential equation (SDE): SDS samples along an SDE trajectory to yield a less noisy sample which then serves as a guidance to optimize a 3D model. However, the randomness in SDE sampling often leads to a diverse and unpredictable sample which is not always less noisy, and thus is not a consistently correct guidance, explaining the vulnerability of SDS. Since for any SDE, there always exists an ordinary differential equation (ODE) whose trajectory sampling can deterministically and consistently converge to the desired target point as the SDE, we propose a novel and effective "Consistent3D" method that explores the ODE deterministic sampling prior for text-to-3D generation. Specifically, at each training iteration, given a rendered image by a 3D model, we first estimate its desired 3D score function by a pre-trained 2D diffusion model, and build an ODE for trajectory sampling. Next, we design a consistency distillation sampling loss which samples along the ODE trajectory to generate two adjacent samples and uses the less noisy sample to guide another more noisy one for distilling the deterministic prior into the 3D model. Experimental results show the efficacy of our Consistent3D in generating high-fidelity and diverse 3D objects and large-scale scenes, as shown in Fig. 1. The codes are available at https://github.com/sail-sg/Consistent3D.
Paper Structure (20 sections, 2 theorems, 23 equations, 7 figures, 4 tables, 1 algorithm)

This paper contains 20 sections, 2 theorems, 23 equations, 7 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Method
  5. Revisit Score Distillation Sampling
  6. Consistency Distillation Sampling
  7. Consistent3D
  8. Experiment
  9. Implementation Details
  10. Text-guided 3D Generation
  11. Comparison with the State-of-The-Art
  12. Ablation Study
  13. Conclusion
  14. Discussion
  15. Experiments
  16. ...and 5 more sections

Key Result

Theorem 1

Assume that the diffusion model $D_{\phi}(\cdot)$ satisfies the Lipschitz condition. Define $\Delta := \sup |t_1 - t_2|$. For any given camera pose $\pi$, if convergence is achieved according to Eq. eq:cds, then there exists a corresponding real image $\mathbf{x}^* \sim p_{\text{data}}(\mathbf{x})$ where $\mathbf{x}_{\pi} = g(\boldsymbol{\theta}, \pi)$ denotes the rendered image for pose $\pi$.

Figures (7)

  • Figure 1: Comparison between the (reverse) trajectory samplings in the stochastic differential equation (SDE) and ordinary differential equation (ODE).
  • Figure 2: Overview of CDS. In each training iteration, the rendered image is perturbed by a fixed noise and then served as a start point of the deterministic flow for computing the CDS loss.
  • Figure 3: Consistent3D can generate diverse and high-fidelity objects or large-scale scenes highly correlated with the given text prompts.
  • Figure 4: Qualitative Comparisons of Text-to-3D Generation. Our approach yields results with enhanced fidelity and more robust geometry.
  • Figure 5: Ablation study of component-wise contribution of Consistent3D: (a) random time step schedule; (b) predetermined time step schedule; (c) random noise in each iteration; (d) our proposed configuration.
  • ...and 2 more figures

Theorems & Definitions (5)

  • Theorem 1
  • proof
  • Theorem 1
  • proof
  • Remark