Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MultiDreamer3D: Multi-concept 3D Customization with Concept-Aware Diffusion Guidance

Wooseok Song, Seunggyu Chang, Jaejun Yoo

TL;DR

MultiDreamer3D tackles multi concept 3D customization by splitting the problem into layout generation and diffusion guided refinement. It uses an LLM driven 3D layout controller to create bounding boxes and a selective point cloud generator to place concept geometry, followed by concept labeling and regional attention driven diffusion updates (CDG with RCA and CISM) to preserve distinct concept identities. The approach addresses object missing and concept mixing, showing strong text and image alignment and favorable user studies across multiple complex scenarios including multiple subjects, property changes, and interactions. This work extends 3D diffusion-based generation to multi concept scenes with coherent layouts and explicit concept attribution, enabling more flexible and controllable 3D content creation.

Abstract

While single-concept customization has been studied in 3D, multi-concept customization remains largely unexplored. To address this, we propose MultiDreamer3D that can generate coherent multi-concept 3D content in a divide-and-conquer manner. First, we generate 3D bounding boxes using an LLM-based layout controller. Next, a selective point cloud generator creates coarse point clouds for each concept. These point clouds are placed in the 3D bounding boxes and initialized into 3D Gaussian Splatting with concept labels, enabling precise identification of concept attributions in 2D projections. Finally, we refine 3D Gaussians via concept-aware interval score matching, guided by concept-aware diffusion. Our experimental results show that MultiDreamer3D not only ensures object presence and preserves the distinct identities of each concept but also successfully handles complex cases such as property change or interaction. To the best of our knowledge, we are the first to address the multi-concept customization in 3D.

MultiDreamer3D: Multi-concept 3D Customization with Concept-Aware Diffusion Guidance

TL;DR

MultiDreamer3D tackles multi concept 3D customization by splitting the problem into layout generation and diffusion guided refinement. It uses an LLM driven 3D layout controller to create bounding boxes and a selective point cloud generator to place concept geometry, followed by concept labeling and regional attention driven diffusion updates (CDG with RCA and CISM) to preserve distinct concept identities. The approach addresses object missing and concept mixing, showing strong text and image alignment and favorable user studies across multiple complex scenarios including multiple subjects, property changes, and interactions. This work extends 3D diffusion-based generation to multi concept scenes with coherent layouts and explicit concept attribution, enabling more flexible and controllable 3D content creation.

Abstract

While single-concept customization has been studied in 3D, multi-concept customization remains largely unexplored. To address this, we propose MultiDreamer3D that can generate coherent multi-concept 3D content in a divide-and-conquer manner. First, we generate 3D bounding boxes using an LLM-based layout controller. Next, a selective point cloud generator creates coarse point clouds for each concept. These point clouds are placed in the 3D bounding boxes and initialized into 3D Gaussian Splatting with concept labels, enabling precise identification of concept attributions in 2D projections. Finally, we refine 3D Gaussians via concept-aware interval score matching, guided by concept-aware diffusion. Our experimental results show that MultiDreamer3D not only ensures object presence and preserves the distinct identities of each concept but also successfully handles complex cases such as property change or interaction. To the best of our knowledge, we are the first to address the multi-concept customization in 3D.
Paper Structure (16 sections, 13 equations, 5 figures, 3 tables)

This paper contains 16 sections, 13 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. 3D Gaussian Splatting
  4. Lifting 2D Diffusion Model to 3D
  5. Method
  6. 3D Layout Generator
  7. Concept-aware Diffusion Guidance
  8. Experiments
  9. Datasets.
  10. Baseline Methods.
  11. Evaluation Metrics.
  12. Qualitative Results.
  13. Quantitative Results.
  14. User Study.
  15. Ablation Study.
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Challenges in multi-concept 3D customization. The 3D content is produced using multi-concept 2D diffusion models using the SDS-based method liang2023luciddreamer. (a) The "C1 car" is missing which leads to poor layout context. (b) The dog's head is combined with a robot toy's body, which we call a concept-mixing problem.
  • Figure 2: Overall pipeline of MultiDreamer3D. (a) The 3D layout controller produces 3D bounding boxes given text descriptions. Subsequently, the selective concept point cloud generator outputs coarse concept point clouds and positions within the 3D bounding boxes. (b) The images and concept masks are rendered from 3D Gaussian Splatting (3DGS) $\Theta$ and updated with concept-aware interval score matching (CISM) loss, facilitated by regional concept attention (RCA).
  • Figure 3: The Regional Concept Attention (RCA) modulates the cross-attention layer in the diffusion model. Individual concept query vectors are computed with image features and each concept masks. Subsequently, key and value vectors for each concept are derived using concept-specific LoRAs and prompts. Then concept-specific attention features are computed with each query, key, and value. The final cross-attention features are aggregated with masked concept-specific attention features.
  • Figure 4: Qualitative results. We compare our method with other baselines in three cases, mulitple subjects, property change, and interaction. The red dashed line indicates the objects mentioned in the text prompt that are missing.
  • Figure 5: Ablation study. (a) generated with baseline (3DGS + FedAVG mcmahan2017communication), (b) with proposed 3D Layout Controller + CISM, (c) combined with Shap-E jun2023shap, (d) combined with pointcloud selection.