S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field

TL;DR

This work tackles realistic 3D indoor scene synthesis by addressing multimodal relationships between scene layout and object details. It introduces Scene Implicit Neural Field (S-INF), which disentangles layout relationships ($l_b$) from detailed object relationships ($f_b$) in a latent space and optimizes them via global distillation and differentiable rendering within an implicit neural field. The approach yields realistic layouts and style-consistent object details, improving retrieval-based synthesis and achieving state-of-the-art results on the 3D-FRONT benchmark. The methodology reduces mode collapse and enhances diversity, facilitating practical ISS applications in VR and interior design through improved realism and visual coherence.

Abstract

Learning-based methods have become increasingly popular in 3D indoor scene synthesis (ISS), showing superior performance over traditional optimization-based approaches. These learning-based methods typically model distributions on simple yet explicit scene representations using generative models. However, due to the oversimplified explicit representations that overlook detailed information and the lack of guidance from multimodal relationships within the scene, most learning-based methods struggle to generate indoor scenes with realistic object arrangements and styles. In this paper, we introduce a new method, Scene Implicit Neural Field (S-INF), for indoor scene synthesis, aiming to learn meaningful representations of multimodal relationships, to enhance the realism of indoor scene synthesis. S-INF assumes that the scene layout is often related to the object-detailed information. It disentangles the multimodal relationships into scene layout relationships and detailed object relationships, fusing them later through implicit neural fields (INFs). By learning specialized scene layout relationships and projecting them into S-INF, we achieve a realistic generation of scene layout. Additionally, S-INF captures dense and detailed object relationships through differentiable rendering, ensuring stylistic consistency across objects. Through extensive experiments on the benchmark 3D-FRONT dataset, we demonstrate that our method consistently achieves state-of-the-art performance under different types of ISS.

Figures (6)

• Figure 1: Unlike previous methods, we enhance the implicit modeling process from the S-INF with scene layout relationships and detailed object relationships, to achieve more layout realistic and style consistency generations.
• Figure 2: Our approach focuses on developing the S-INF to enable efficient capture of multimodal relationships and generate realistic and reliable 3D indoor scenes. We utilize the scene encoder to distill the realistic multimodal relationships into the S-INF. We also use differentiable rendering to enhance the S-INF style consistency information in detailed style relationships. Those optimized equip the S-INF with genuine multimodal relationship understanding capabilities, facilitating the generation of realistic, and style-invariant 3D indoor scenes.
• Figure 3: Integration between scene layout relationships and detailed object relationships. We leverage a specialized scene encoder to construct scene layout relationships (and little detailed object relationships) and distill multimodal relationships into S-INF. On the other hand, the rendered image provides dense and detailed style information, enriching the detailed object relationships in S-INF.
• Figure 4: Qualitative comparisons of bedroom with different baselines on ISS.
• Figure 5: Realistic ISS from multimodal relationships.