DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priors

TL;DR

DreamDissector tackles the challenge of generating multiple independently controllable 3D objects with interactions from a single text-to-3D NeRF. It introduces Neural Category Field (NeCF) to disentangle a NeRF into sub-NeRFs and Category Score Distillation Sampling (CSDS) guided by Deep Concept Mining (DCM) to align per-object concepts with diffusion priors; sub-NeRFs are then refined as DMTets to yield textured meshes. The approach achieves improved geometry and textures, supports per-object editing and replacement, and outperforms baselines in CLIP-based metrics and qualitative assessments. With efficient training times (roughly minutes for NeCF/DCM stages and thousands of steps for refinement), DreamDissector offers a practical, object-level control framework for multi-object 3D synthesis and editing in creative pipelines.

Abstract

Text-to-3D generation has recently seen significant progress. To enhance its practicality in real-world applications, it is crucial to generate multiple independent objects with interactions, similar to layer-compositing in 2D image editing. However, existing text-to-3D methods struggle with this task, as they are designed to generate either non-independent objects or independent objects lacking spatially plausible interactions. Addressing this, we propose DreamDissector, a text-to-3D method capable of generating multiple independent objects with interactions. DreamDissector accepts a multi-object text-to-3D NeRF as input and produces independent textured meshes. To achieve this, we introduce the Neural Category Field (NeCF) for disentangling the input NeRF. Additionally, we present the Category Score Distillation Sampling (CSDS), facilitated by a Deep Concept Mining (DCM) module, to tackle the concept gap issue in diffusion models. By leveraging NeCF and CSDS, we can effectively derive sub-NeRFs from the original scene. Further refinement enhances geometry and texture. Our experimental results validate the effectiveness of DreamDissector, providing users with novel means to control 3D synthesis at the object level and potentially opening avenues for various creative applications in the future.

Figures (20)

• Figure 1: Results and applications of our DreamDissector.Top left: DreamDissector can generate multiple independent textured meshes with plausible interactions. Top right: Facilitating text-guided texturing at the object level. Bottom left: Enhancing convenient manual user geometry editing through simple operations. Bottom right: Facilitating text-guided controllable object replacement.
• Figure 2: Method overview. We generate multiple independent interactive 3D objects in a coarse-to-fine manner. Initially, we render a view of the input text-to-3D NeRF for Deep Concept Mining (DCM), obtaining both the T2I diffusion model and the corresponding text embedding. We then use the mined embedding and the T2I diffusion model to train the neural category field (NeCF) using category score distillation sampling (CSDS). After disentangling the input NeRF, we convert the sub-NeRFs into DMTets and fine-tune these for further refinement. Finally, we export independent surface meshes with improved geometries and textures.
• Figure 3: Left: Concept discrepancy in diffusion models. The text prompt is "A chimpanzee looking through a telescope". Right: Overview of Deep Concept Mining (DCM). We finetune the text embedding and the T2I diffusion model with the masked diffusion loss (Eq. \ref{['eq:masked_diffusion']}).
• Figure 4: Qualitative Results. The text prompts used to generate the NeRF are available in the supplementary file.
• Figure 5: Comparison with two baselines. We show the independent objects for ease of comparison. The composed objects and more comparisons are available in the supplementary file.