Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior

Cheng Chen, Xiaofeng Yang, Fan Yang, Chengzeng Feng, Zhoujie Fu, Chuan-Sheng Foo, Guosheng Lin, Fayao Liu

TL;DR

A new framework Sculpt3D is presented that equips the current pipeline with explicit injection of 3D priors from retrieved reference objects without re-training the 2D diffusion model and demonstrates that high-quality and diverse 3D geometry can be guaranteed by keypoints supervision through a sparse ray sampling approach.

Abstract

Recent works on text-to-3d generation show that using only 2D diffusion supervision for 3D generation tends to produce results with inconsistent appearances (e.g., faces on the back view) and inaccurate shapes (e.g., animals with extra legs). Existing methods mainly address this issue by retraining diffusion models with images rendered from 3D data to ensure multi-view consistency while struggling to balance 2D generation quality with 3D consistency. In this paper, we present a new framework Sculpt3D that equips the current pipeline with explicit injection of 3D priors from retrieved reference objects without re-training the 2D diffusion model. Specifically, we demonstrate that high-quality and diverse 3D geometry can be guaranteed by keypoints supervision through a sparse ray sampling approach. Moreover, to ensure accurate appearances of different views, we further modulate the output of the 2D diffusion model to the correct patterns of the template views without altering the generated object's style. These two decoupled designs effectively harness 3D information from reference objects to generate 3D objects while preserving the generation quality of the 2D diffusion model. Extensive experiments show our method can largely improve the multi-view consistency while retaining fidelity and diversity. Our project page is available at: https://stellarcheng.github.io/Sculpt3D/.

Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior

TL;DR

A new framework Sculpt3D is presented that equips the current pipeline with explicit injection of 3D priors from retrieved reference objects without re-training the 2D diffusion model and demonstrates that high-quality and diverse 3D geometry can be guaranteed by keypoints supervision through a sparse ray sampling approach.

Abstract

Recent works on text-to-3d generation show that using only 2D diffusion supervision for 3D generation tends to produce results with inconsistent appearances (e.g., faces on the back view) and inaccurate shapes (e.g., animals with extra legs). Existing methods mainly address this issue by retraining diffusion models with images rendered from 3D data to ensure multi-view consistency while struggling to balance 2D generation quality with 3D consistency. In this paper, we present a new framework Sculpt3D that equips the current pipeline with explicit injection of 3D priors from retrieved reference objects without re-training the 2D diffusion model. Specifically, we demonstrate that high-quality and diverse 3D geometry can be guaranteed by keypoints supervision through a sparse ray sampling approach. Moreover, to ensure accurate appearances of different views, we further modulate the output of the 2D diffusion model to the correct patterns of the template views without altering the generated object's style. These two decoupled designs effectively harness 3D information from reference objects to generate 3D objects while preserving the generation quality of the 2D diffusion model. Extensive experiments show our method can largely improve the multi-view consistency while retaining fidelity and diversity. Our project page is available at: https://stellarcheng.github.io/Sculpt3D/.
Paper Structure (22 sections, 7 equations, 17 figures, 1 table)

This paper contains 22 sections, 7 equations, 17 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Approach
  4. Revisiting 2D Diffusion for 3D Generation
  5. 3D and 2D Co-supervised 3D Generation
  6. 3D Prior Guided Shape Learning
  7. 3D Appearance Modulated 2D Diffusion Prior
  8. Experiments
  9. Implementation Details
  10. Results of Sculpt3D
  11. Comparison to Baselines
  12. Quantitative Evaluation
  13. Ablation Study
  14. Conclusion & Limitation
  15. Supplementary Materials
  16. ...and 7 more sections

Figures (17)

  • Figure 1: Comparison of objects generated by our method and ProlificDreamer. We retain the 2D model's capability to produce high-fidelity objects and adaptively learn 3D information from reference templates retrieved from external datasets.
  • Figure 2: Our methods can generate high-fidelity objects with decent shapes using various text prompts. The model adaptively incorporates information from the reference shape displayed on the left, resulting in the creation of objects that range from moderately resembling to substantially diverging from the reference shape. Please find more video results in the supplementary materials.
  • Figure 3: As shown in the first row, our method can generate diverse 3D objects given the same reference shape. The second row also shows the diverse results generated by randomly selecting reference objects from the top five retrieved samples. All templates are marked as gray and shown in the corner.
  • Figure 4: Given a text prompt, we retrieve the most semantically matching samples from an external 3D database. With the retrieved object, we sparsely select the keypoints of the reference shape to co-supervise the geometry with 2D diffusion model. The appearance of the reference object is also used to modulate the 2D diffusion to avoid appearance ambiguity.
  • Figure 5: Illustration of the appearance modulation. Four canonical views of the templates are transferred to the generated object's style to modulate the 2D diffusion.
  • ...and 12 more figures