Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Controllable 3D Object Generation with Single Image Prompt

Jaeseok Lee, Jaekoo Lee

TL;DR

This work tackles controllable 3D object generation from a single image without relying on textual inversion. It introduces Controllable Image Prompt Score Distillation Sampling (IPSDS) that uses an IP-Adapter and ControlNet conditioned on depth estimates, along with a depth-conditioned warmup strategy to stabilize 3D consistency within a two-stage NeRF-DMTet pipeline. The approach achieves competitive quantitative performance with state-of-the-art methods and demonstrates improved 3D consistency and fidelity in qualitative evaluations, validated by a user study. By eliminating the need for pseudo-text prompts and enabling depth/pose/sketch conditioning, it offers faster training and practical utility for image-to-3D generation in AR/VR workflows, with code released at GitHub.

Abstract

Recently, the impressive generative capabilities of diffusion models have been demonstrated, producing images with remarkable fidelity. Particularly, existing methods for the 3D object generation tasks, which is one of the fastest-growing segments in computer vision, pre-dominantly use text-to-image diffusion models with textual inversion which train a pseudo text prompt to describe the given image. In practice, various text-to-image generative models employ textual inversion to learn concepts or styles of target object in the pseudo text prompt embedding space, thereby generating sophisticated outputs. However, textual inversion requires additional training time and lacks control ability. To tackle this issues, we propose two innovative methods: (1) using an off-the-shelf image adapter that generates 3D objects without textual inversion, offering enhanced control over conditions such as depth, pose, and text. (2) a depth conditioned warmup strategy to enhance 3D consistency. In experimental results, ours show qualitatively and quantitatively comparable performance and improved 3D consistency to the existing text-inversion-based alternatives. Furthermore, we conduct a user study to assess (i) how well results match the input image and (ii) whether 3D consistency is maintained. User study results show that our model outperforms the alternatives, validating the effectiveness of our approaches. Our code is available at GitHub repository:https://github.com/Seooooooogi/Control3D_IP/

Controllable 3D Object Generation with Single Image Prompt

TL;DR

This work tackles controllable 3D object generation from a single image without relying on textual inversion. It introduces Controllable Image Prompt Score Distillation Sampling (IPSDS) that uses an IP-Adapter and ControlNet conditioned on depth estimates, along with a depth-conditioned warmup strategy to stabilize 3D consistency within a two-stage NeRF-DMTet pipeline. The approach achieves competitive quantitative performance with state-of-the-art methods and demonstrates improved 3D consistency and fidelity in qualitative evaluations, validated by a user study. By eliminating the need for pseudo-text prompts and enabling depth/pose/sketch conditioning, it offers faster training and practical utility for image-to-3D generation in AR/VR workflows, with code released at GitHub.

Abstract

Recently, the impressive generative capabilities of diffusion models have been demonstrated, producing images with remarkable fidelity. Particularly, existing methods for the 3D object generation tasks, which is one of the fastest-growing segments in computer vision, pre-dominantly use text-to-image diffusion models with textual inversion which train a pseudo text prompt to describe the given image. In practice, various text-to-image generative models employ textual inversion to learn concepts or styles of target object in the pseudo text prompt embedding space, thereby generating sophisticated outputs. However, textual inversion requires additional training time and lacks control ability. To tackle this issues, we propose two innovative methods: (1) using an off-the-shelf image adapter that generates 3D objects without textual inversion, offering enhanced control over conditions such as depth, pose, and text. (2) a depth conditioned warmup strategy to enhance 3D consistency. In experimental results, ours show qualitatively and quantitatively comparable performance and improved 3D consistency to the existing text-inversion-based alternatives. Furthermore, we conduct a user study to assess (i) how well results match the input image and (ii) whether 3D consistency is maintained. User study results show that our model outperforms the alternatives, validating the effectiveness of our approaches. Our code is available at GitHub repository:https://github.com/Seooooooogi/Control3D_IP/

Paper Structure

This paper contains 13 sections, 7 equations, 10 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Controllable Image Prompt Score Distillation Sampling
  5. Depth Conditioned Warmup Strategy
  6. Experiments
  7. Implementation Details
  8. Quantitative Results
  9. Qualitative Results
  10. Discussion
  11. User Study
  12. Conclusion
  13. Acknowledgements

Figures (10)

  • Figure 1: Comparison between (a) existing method using text-guided priors via textual inversion and (b) our method that employs a single image prompt.
  • Figure 2: Overall architecture of our proposed 3D generative model. During training, our method iteratively use the following stages: (i) controllable image prompt score distillation sampling, and (ii) depth conditioned warmup strategy.
  • Figure 3: An overview of controllable image prompt score distillation sampling. See details on Sec \ref{['sec1']}.
  • Figure 4: An overview of depth conditioned warmup strategy. See details on Sec \ref{['sec2']}.
  • Figure 5: Qualitative image-to-3D generation performance comparison with SOTA alternatives (Dreamfusion poole2022dreamfusion, RealFusion melaskyriazi2023realfusion, LGM tang2024lgm, LRM hong2023lrm
  • ...and 5 more figures