ReSpace: Text-Driven 3D Indoor Scene Synthesis and Editing with Preference Alignment

TL;DR

ReSpace tackles the challenge of text-driven 3D indoor scene synthesis and editing by introducing a Structured Scene Representation and a specialized SG-LLM trained with SFT and GRPO. It decouples asset selection from scene description for asset-agnostic deployment and uses a zero-shot LLM for removals and prompt generation, enabling efficient object addition and full-scene synthesis. A novel voxelization-based loss (VBL) provides fine-grained geometry-aware evaluation and serves as a verifiable reward during preference alignment. Experimental results on non-rectangular layouts show state-of-the-art performance for object addition and strong human-perceived quality for full scenes, with practical implications for editable, inventory-agnostic interior design and robotics contexts.

Abstract

Scene synthesis and editing has emerged as a promising direction in computer graphics. Current trained approaches for 3D indoor scenes either oversimplify object semantics through one-hot class encodings (e.g., 'chair' or 'table'), require masked diffusion for editing, ignore room boundaries, or rely on floor plan renderings that fail to capture complex layouts. LLM-based methods enable richer semantics via natural language (e.g., 'modern studio with light wood furniture'), but lack editing functionality, are limited to rectangular layouts, or rely on weak spatial reasoning from implicit world models. We introduce ReSpace, a generative framework for text-driven 3D indoor scene synthesis and editing using autoregressive language models. Our approach features a compact structured scene representation with explicit room boundaries that enables asset-agnostic deployment and frames scene editing as a next-token prediction task. We leverage a dual-stage training approach combining supervised fine-tuning and preference alignment, enabling a specially trained language model for object addition that accounts for user instructions, spatial geometry, object semantics, and scene-level composition. For scene editing, we employ a zero-shot LLM to handle object removal and prompts for addition. We further introduce a voxelization-based evaluation capturing fine-grained geometry beyond 3D bounding boxes. Experimental results surpass state-of-the-art on addition and achieve superior human-perceived quality on full scene synthesis.

Figures (18)

• Figure 1: We introduce a novel text-driven framework for 3D indoor scene synthesis, completion, and editing—supporting object addition, removal, and swapping via natural language prompts.
• Figure 2: ReSpace Overview. Given a user instruction via text and an existing scene represented via SSR, we autoregressively perform 3D scene synthesis and editing. A zero-shot LLM converts user instructions to sequential commands for object removal and addition, with the latter done via specialized SG-LLM ($p_\theta$) and removal via zero-shot SSR editing.
• Figure 3: Example of description and prompt bank generation for a single asset in the catalog.
• Figure 4: Scene represented with 3D bounding boxes in blue and bounds in red (A), with 3D assets (B), their voxelized counterpart (C), and some examples of OOB/MBL voxel violations (D). Note how the desk and chair interact smoothly in mesh space compared to their blue bounding boxes, while the lamp is largely OOB with its bounding box but only minor with its mesh.
• Figure 5: Qualitative results on single instructions, with our method performing the strongest. For ours, we use $\text{ReSpace/A}^{\dagger}$. We show a failure case on the last row where all methods perform poorly.