SceneNAT: Masked Generative Modeling for Language-Guided Indoor Scene Synthesis

TL;DR

SceneNAT tackles language-conditioned 3D indoor scene synthesis by introducing a masked non-autoregressive transformer (MISM) that reconstructs entire scenes in parallel. It combines a scene decoder with a dedicated triplet predictor for explicit relational reasoning, trained with a reconstruction loss and a set-based bipartite-matching objective. On augmented 3D-FRONT data, it achieves state-of-the-art results in instruction adherence and spatial accuracy while being significantly more efficient than autoregressive or diffusion baselines. This approach enables real-time, controllable generation and robust generalization to complex relational instructions, with potential extensions to multimodal conditioning and generative asset creation.

Abstract

We present SceneNAT, a single-stage masked non-autoregressive Transformer that synthesizes complete 3D indoor scenes from natural language instructions through only a few parallel decoding passes, offering improved performance and efficiency compared to prior state-of-the-art approaches. SceneNAT is trained via masked modeling over fully discretized representations of both semantic and spatial attributes. By applying a masking strategy at both the attribute level and the instance level, the model can better capture intra-object and inter-object structure. To boost relational reasoning, SceneNAT employs a dedicated triplet predictor for modeling the scene's layout and object relationships by mapping a set of learnable relation queries to a sparse set of symbolic triplets (subject, predicate, object). Extensive experiments on the 3D-FRONT dataset demonstrate that SceneNAT achieves superior performance compared to state-of-the-art autoregressive and diffusion baselines in both semantic compliance and spatial arrangement accuracy, while operating with substantially lower computational cost.

Figures (25)

• Figure 1: An illustration of the proposed method for text-driven 3D indoor scene generation. SceneNAT supports a wide range of applications, including complex instruction following, object rearrangement, recommendation, and stylization. Given a text instruction, our model iteratively predicts the attributes (class, appearance, layout) for a set of initially masked objects in a non-autoregressive manner. The resulting set of object attributes guides the retrieval and placement of 3D assets to construct the scene.
• Figure 2: An overview of the SceneNAT framework. To achieve both visual quality and controllability, our model employs two specialized components. A scene decoder uses masked modeling to generate a globally coherent scene structure. Concurrently, a triplet predictor transforms a set of learnable triplet queries into specific relational constraints parsed from the text instruction. The intermediate triplet features are fused into the layout decoder via cross-attention layers to generate the final indoor scene attributes.
• Figure 3: Comparison of qualitative results on language-guided scene generation. SceneNAT demonstrates superior performance in generating realistic 3D scenes that faithfully adhere to complex instructions. Additional qualitative results can be found in Appendix \ref{['qualitative_appendix']}.
• Figure 4: Comparison of iRecall with respect to the number of relational constraints in the instructions.
• Figure 5: Comparison of FID and iRecall with respect to the number of inference steps.