Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation

Yang Chen, Yingwei Pan, Haibo Yang, Ting Yao, Tao Mei

TL;DR

This work introduces a novel Visual Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the visual appearance knowledge in 2D visual prompt to boost text-to-3D generation and significantly eases the learning of visual appearance of 3D models and thus leads to higher visual fidelity with more detailed textures.

Abstract

Recent innovations on text-to-3D generation have featured Score Distillation Sampling (SDS), which enables the zero-shot learning of implicit 3D models (NeRF) by directly distilling prior knowledge from 2D diffusion models. However, current SDS-based models still struggle with intricate text prompts and commonly result in distorted 3D models with unrealistic textures or cross-view inconsistency issues. In this work, we introduce a novel Visual Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the visual appearance knowledge in 2D visual prompt to boost text-to-3D generation. Instead of solely supervising SDS with text prompt, VP3D first capitalizes on 2D diffusion model to generate a high-quality image from input text, which subsequently acts as visual prompt to strengthen SDS optimization with explicit visual appearance. Meanwhile, we couple the SDS optimization with additional differentiable reward function that encourages rendering images of 3D models to better visually align with 2D visual prompt and semantically match with text prompt. Through extensive experiments, we show that the 2D Visual Prompt in our VP3D significantly eases the learning of visual appearance of 3D models and thus leads to higher visual fidelity with more detailed textures. It is also appealing in view that when replacing the self-generating visual prompt with a given reference image, VP3D is able to trigger a new task of stylized text-to-3D generation. Our project page is available at https://vp3d-cvpr24.github.io.

VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation

TL;DR

This work introduces a novel Visual Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the visual appearance knowledge in 2D visual prompt to boost text-to-3D generation and significantly eases the learning of visual appearance of 3D models and thus leads to higher visual fidelity with more detailed textures.

Abstract

Recent innovations on text-to-3D generation have featured Score Distillation Sampling (SDS), which enables the zero-shot learning of implicit 3D models (NeRF) by directly distilling prior knowledge from 2D diffusion models. However, current SDS-based models still struggle with intricate text prompts and commonly result in distorted 3D models with unrealistic textures or cross-view inconsistency issues. In this work, we introduce a novel Visual Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the visual appearance knowledge in 2D visual prompt to boost text-to-3D generation. Instead of solely supervising SDS with text prompt, VP3D first capitalizes on 2D diffusion model to generate a high-quality image from input text, which subsequently acts as visual prompt to strengthen SDS optimization with explicit visual appearance. Meanwhile, we couple the SDS optimization with additional differentiable reward function that encourages rendering images of 3D models to better visually align with 2D visual prompt and semantically match with text prompt. Through extensive experiments, we show that the 2D Visual Prompt in our VP3D significantly eases the learning of visual appearance of 3D models and thus leads to higher visual fidelity with more detailed textures. It is also appealing in view that when replacing the self-generating visual prompt with a given reference image, VP3D is able to trigger a new task of stylized text-to-3D generation. Our project page is available at https://vp3d-cvpr24.github.io.
Paper Structure (14 sections, 8 equations, 5 figures, 1 table)

This paper contains 14 sections, 8 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. VP3D
  4. Background
  5. Visual-prompted Score Distillation Sampling
  6. Learning with Reward Feedback
  7. 3D Representation and Training
  8. Experiments
  9. Experimental Settings
  10. Quantitative Results
  11. Qualitative Results
  12. Ablation Study
  13. Extension to Stylized Text-to-3D Generation
  14. Conclusion

Figures (5)

  • Figure 1: Exisiting text-to-3D generation techniques (e.g., Magic3D lin2022magic3d and ProlificDreamer wang2023prolificdreamer) often suffer from degenerated results (e.g., over-saturated appearances or inaccurate geometries). Our VP3D novelly integrates a visual prompt to strength score distillation sampling, leading to better 3D results.
  • Figure 2: An overview of the proposed VP3D framework for visual prompted text-to-3D generation.
  • Figure 3: Comparisons on qualitative results of our VP3D with other text-to-3D techniques on T$^3$Bench he2023t. The prompts are (a) "A fuzzy pink flamingo lawn ornament", (b) "A blooming potted orchid with purple flowers", (c) "A blue butterfly on a pink flower",(d) "A lighthouse on a rocky shore",(e) "Hot popcorn jump out from the red striped popcorn maker",(f) "A chef is making pizza dough in the kitchen". (a-b), (c-d), (e-f) belongs to the Single Object, Single Object with Surr and Multi Objects category in T$^3$Bench, respectively.
  • Figure 4: Stylized text-to-3D generation results of our VP3D.
  • Figure 5: Comparisons on qualitative results by using different ablated runs of our VP3D. The text prompts are (a) "A broken tablespoon lies next to an empty sugar bowl" and (b) "A chameleon perched on a tree branch".