Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating Images with 3D Annotations Using Diffusion Models

Wufei Ma, Qihao Liu, Jiahao Wang, Angtian Wang, Xiaoding Yuan, Yi Zhang, Zihao Xiao, Guofeng Zhang, Beijia Lu, Ruxiao Duan, Yongrui Qi, Adam Kortylewski, Yaoyao Liu, Alan Yuille

TL;DR

This work tackles the lack of explicit 3D control in diffusion-based image generation by presenting 3D-DST, which injects 3D geometry via 3D visual prompts derived from rendering edge maps of CAD models and diverse text prompts generated by LLMs. By integrating these prompts with ControlNet and diffusion models, the method can produce photorealistic images whose underlying 3D structure is controllable, enabling automatic ground-truth 3D annotations. The authors demonstrate strong improvements across image classification, 3D pose estimation, and 3D object detection in both ID and OOD settings on multiple benchmarks, including ImageNet variants and PASCAL3D+/ObjectNet3D, with notable gains such as up to $3.8$ percentage points on ImageNet-100 using DeiT-B and larger benefits in downstream 3D tasks. These results highlight the practical impact of leveraging explicit 3D conditioning and diverse prompts to bolster robustness and 3D understanding in vision systems.

Abstract

Diffusion models have emerged as a powerful generative method, capable of producing stunning photo-realistic images from natural language descriptions. However, these models lack explicit control over the 3D structure in the generated images. Consequently, this hinders our ability to obtain detailed 3D annotations for the generated images or to craft instances with specific poses and distances. In this paper, we propose 3D Diffusion Style Transfer (3D-DST), which incorporates 3D geometry control into diffusion models. Our method exploits ControlNet, which extends diffusion models by using visual prompts in addition to text prompts. We generate images of the 3D objects taken from 3D shape repositories (e.g., ShapeNet and Objaverse), render them from a variety of poses and viewing directions, compute the edge maps of the rendered images, and use these edge maps as visual prompts to generate realistic images. With explicit 3D geometry control, we can easily change the 3D structures of the objects in the generated images and obtain ground-truth 3D annotations automatically. This allows us to improve a wide range of vision tasks, e.g., classification and 3D pose estimation, in both in-distribution (ID) and out-of-distribution (OOD) settings. We demonstrate the effectiveness of our method through extensive experiments on ImageNet-100/200, ImageNet-R, PASCAL3D+, ObjectNet3D, and OOD-CV. The results show that our method significantly outperforms existing methods, e.g., 3.8 percentage points on ImageNet-100 using DeiT-B.

Generating Images with 3D Annotations Using Diffusion Models

TL;DR

This work tackles the lack of explicit 3D control in diffusion-based image generation by presenting 3D-DST, which injects 3D geometry via 3D visual prompts derived from rendering edge maps of CAD models and diverse text prompts generated by LLMs. By integrating these prompts with ControlNet and diffusion models, the method can produce photorealistic images whose underlying 3D structure is controllable, enabling automatic ground-truth 3D annotations. The authors demonstrate strong improvements across image classification, 3D pose estimation, and 3D object detection in both ID and OOD settings on multiple benchmarks, including ImageNet variants and PASCAL3D+/ObjectNet3D, with notable gains such as up to percentage points on ImageNet-100 using DeiT-B and larger benefits in downstream 3D tasks. These results highlight the practical impact of leveraging explicit 3D conditioning and diverse prompts to bolster robustness and 3D understanding in vision systems.

Abstract

Diffusion models have emerged as a powerful generative method, capable of producing stunning photo-realistic images from natural language descriptions. However, these models lack explicit control over the 3D structure in the generated images. Consequently, this hinders our ability to obtain detailed 3D annotations for the generated images or to craft instances with specific poses and distances. In this paper, we propose 3D Diffusion Style Transfer (3D-DST), which incorporates 3D geometry control into diffusion models. Our method exploits ControlNet, which extends diffusion models by using visual prompts in addition to text prompts. We generate images of the 3D objects taken from 3D shape repositories (e.g., ShapeNet and Objaverse), render them from a variety of poses and viewing directions, compute the edge maps of the rendered images, and use these edge maps as visual prompts to generate realistic images. With explicit 3D geometry control, we can easily change the 3D structures of the objects in the generated images and obtain ground-truth 3D annotations automatically. This allows us to improve a wide range of vision tasks, e.g., classification and 3D pose estimation, in both in-distribution (ID) and out-of-distribution (OOD) settings. We demonstrate the effectiveness of our method through extensive experiments on ImageNet-100/200, ImageNet-R, PASCAL3D+, ObjectNet3D, and OOD-CV. The results show that our method significantly outperforms existing methods, e.g., 3.8 percentage points on ImageNet-100 using DeiT-B.
Paper Structure (52 sections, 6 equations, 11 figures, 8 tables, 1 algorithm)

This paper contains 52 sections, 6 equations, 11 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Synthetic data augmentation.
  4. Diffusion models
  5. Large language models (LLM).
  6. 3D Diffusion Style Transfer (3D-DST)
  7. Background: Diffusion Models with Text and 2D Visual Prompts
  8. Diffusion model pipeline.
  9. Adding text prompts via cross-attention.
  10. Adding 2D visual prompts via ControlNet.
  11. Limitations of existing methods.
  12. 3D Visual Prompts by Graphics-based Rendering
  13. Graphics-based rendering.
  14. Edge map computation.
  15. Diverse Text Prompts by LLM
  16. ...and 37 more sections

Figures (11)

  • Figure 1: (a) Visualization of our 3D-DST. Our proposed solution, 3D-DST, leverages both 3D visual prompts and large language model (LLM) text prompts to generate diverse images from a CAD model. The use of 3D visual prompts enables explicit control over the 3D structure of the object within the generated images, such as varying 3D poses and distances. On the other hand, LLM text prompts facilitate the automatic generation of images with diverse backgrounds, weather conditions, and colors. (b) Performance comparison. Our model can be utilized to generate data to enhance both in-distribution (ID) and out-of-distribution (OOD) performance. We report results of DeiT-S on ImageNet-$100$tian2020contrastive and ImageNet-R hendrycks2021many. "Text-to-Image" denotes using diffusion models without 3D control to augment the data he2023is.
  • Figure 2: Our 3D-DST comprises three essential steps. (1) 3D visual prompt generation. We generate images of 3D objects taken from a 3D shape repository (e.g., ShapeNet and Objaverse), render them from a variety of viewpoints and distances, compute the edge maps of the rendered images, and use these edge maps as 3D visual prompts. (2) Text prompt generation. Our approach involves combining the class names of objects with the associated tags or keywords of the CAD models. This combined information forms the initial text prompts. Then, we enhance these prompts by incorporating the descriptions generated by LLaMA. (3) Image generation. We generate photo-realistic images with 3D visual and text prompts using Stable Diffusion and ControlNet.
  • Figure 3: Left: Visualizations of 3D-DST synthetic data for image classification and 3D pose estimation. Right: Objects with various appearances can be generated from only a limited number of CAD models by conditioning on diverse textual prompts.
  • Figure 4: Qualitative examples of using different types of 3D control. We experimented with three different types of 3D control: edge maps (top), MiDaS predicted depth (middle), and Blender rendered depth (bottom), using the same 3D model and text prompts. Qualitative results show that using edge maps as 3D control gives overall better outputs.
  • Figure 5: Human evaluation results. We collect feedback from human evaluators and find that about 75% of the generated 3D-DST images are consistent with both the textual and visual prompts. By analyzing the failure cases, we identify a limitation of our model to be images with challenging and uncommon viewpoints.
  • ...and 6 more figures