Mixed Diffusion for 3D Indoor Scene Synthesis

TL;DR

The paper tackles automatic floor-plan conditioned 3D indoor scene synthesis.It introduces MiDiffusion, a mixed discrete-continuous diffusion model that jointly models object categories and geometry, using independent domain corruption and a fused DDPM-D3PM loss.A time-variant transformer denoiser conditioned on floor plans and a corruption-masking strategy for partial constraints enable effective scene completion and furniture arrangement without retraining.Evaluations on 3D-FRONT show MiDiffusion achieves superior realism and boundary adherence compared with autoregressive and diffusion baselines, with well-supported ablations and discussion of limitations.

Abstract

Generating realistic 3D scenes is an area of growing interest in computer vision and robotics. However, creating high-quality, diverse synthetic 3D content often requires expert intervention, making it costly and complex. Recently, efforts to automate this process with learning techniques, particularly diffusion models, have shown significant improvements in tasks like furniture rearrangement. However, applying diffusion models to floor-conditioned indoor scene synthesis remains under-explored. This task is especially challenging as it requires arranging objects in continuous space while selecting from discrete object categories, posing unique difficulties for conventional diffusion methods. To bridge this gap, we present MiDiffusion, a novel mixed discrete-continuous diffusion model designed to synthesize plausible 3D indoor scenes given a floor plan and pre-arranged objects. We represent a scene layout by a 2D floor plan and a set of objects, each defined by category, location, size, and orientation. Our approach uniquely applies structured corruption across mixed discrete semantic and continuous geometric domains, resulting in a better-conditioned problem for denoising. Evaluated on the 3D-FRONT dataset, MiDiffusion outperforms state-of-the-art autoregressive and diffusion models in floor-conditioned 3D scene synthesis. Additionally, it effectively handles partial object constraints via a corruption-and-masking strategy without task-specific training, demonstrating advantages in scene completion and furniture arrangement tasks.

Figures (10)

• Figure 1: Denoising network. The time-variant decoder backbone takes object features as input, conditioned on the floor plan feature. We utilize the time-variant transformer decoder block from VQ-Diffusion Gu22cvpr-VQ-Diffusion.
• Figure 2: Floor-conditioned scene synthesis. The meshes are retrieved from the 3D-FUTURE Fu21ijcv-3d-future dataset by size matching within predicted semantic category. MiDiffusion generates realistic arrangements while respecting boundary constraints.
• Figure 3: Scene completion. Bedroom (top) and living room (bottom) scene completion examples.
• Figure 4: Furniture arrangement. Different table-chair arrangements (top) and different furniture placement directions (bottom).
• Figure 5: Example bedroom top-down orthographic projection images for quantitative evaluations.