MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation

TL;DR

MIDI tackles the challenge of generating coherent multi-object 3D scenes from a single image by extending pre-trained image-to-3D object diffusion models to a multi-instance diffusion framework. A key contribution is the multi-instance attention mechanism, which enables cross-object interactions and global scene coherence during end-to-end generation, conditioned on both global scene context and local object inputs. The approach is trained with limited scene-level data while regularizing with single-object data to preserve generalization, and it achieves state-of-the-art results on synthetic, real, and stylized datasets. This work offers a scalable, efficient pathway to high-quality 3D scene generation with strong generalization across diverse inputs and layouts.

Abstract

This paper introduces MIDI, a novel paradigm for compositional 3D scene generation from a single image. Unlike existing methods that rely on reconstruction or retrieval techniques or recent approaches that employ multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D object generation models to multi-instance diffusion models, enabling the simultaneous generation of multiple 3D instances with accurate spatial relationships and high generalizability. At its core, MIDI incorporates a novel multi-instance attention mechanism, that effectively captures inter-object interactions and spatial coherence directly within the generation process, without the need for complex multi-step processes. The method utilizes partial object images and global scene context as inputs, directly modeling object completion during 3D generation. During training, we effectively supervise the interactions between 3D instances using a limited amount of scene-level data, while incorporating single-object data for regularization, thereby maintaining the pre-trained generalization ability. MIDI demonstrates state-of-the-art performance in image-to-scene generation, validated through evaluations on synthetic data, real-world scene data, and stylized scene images generated by text-to-image diffusion models.

Figures (11)

• Figure 1: MIDI generates compositional 3D scenes from a single image by extending pre-trained image-to-3D object generation models to multi-instance diffusion models, incorporating a novel multi-instance attention mechanism that captures inter-object interactions. (a) shows our generated scenes compared with those reconstructed by existing methods. (b) presents our generated results on synthetic data, real-world images, and stylized images.
• Figure 2: Comparison between our scene generation pipeline with multi-instance diffusion and existing compositional generation methods.
• Figure 3: Method overview. Based on 3D object generation models, MIDI denoises the latent representations of multiple 3D instances simultaneously using a weight-shared DiT module. The multi-instance attention layers are introduced to learn cross-instance interaction and enable global awareness, while cross-attention layers integrate the information of object images and global scene context.
• Figure 4: Multi-instance attention. We extend the original object self-attention, where tokens of each object query only themselves, to multi-instance attention, where tokens of each instance query all tokens from all instances in the scene.
• Figure 5: Qualitative comparisons on synthetic datasets, including 3D-Front fu20213dfront and BlendSwap azinovic2022neuralrgbd.