Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing

Haoyu Zhen, Xiaolong Li, Yilin Zhao, Han Zhang, Sifei Liu, Kaichun Mo, Chuang Gan, Subhashree Radhakrishnan

Abstract

Large Language Models (LLMs) and Vision Language Models (VLMs) have shown impressive reasoning abilities, yet they struggle with spatial understanding and layout consistency when performing fine-grained visual editing. We introduce a Structured Reasoning framework that performs text-conditioned spatial layout editing via scene-graph reasoning. Given an input scene graph and a natural-language instruction, the model reasons over the graph to generate an updated scene graph that satisfies the text condition while maintaining spatial coherence. By explicitly guiding the reasoning process through structured relational representations, our approach improves both interpretability and control over spatial relationships. We evaluate our method on a new text-guided layout editing benchmark encompassing sorting, spatial alignment, and room-editing tasks. Our training paradigm yields an average 15% improvement in IoU and 25% reduction in center-distance error compared to Chain of Thought Fine-tuning (CoT-SFT) and vanilla GRPO baselines. Compared to SOTA zero-shot LLMs, our best models achieve up to 20% higher mIoU, demonstrating markedly improved spatial precision.

3D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing

Abstract

Large Language Models (LLMs) and Vision Language Models (VLMs) have shown impressive reasoning abilities, yet they struggle with spatial understanding and layout consistency when performing fine-grained visual editing. We introduce a Structured Reasoning framework that performs text-conditioned spatial layout editing via scene-graph reasoning. Given an input scene graph and a natural-language instruction, the model reasons over the graph to generate an updated scene graph that satisfies the text condition while maintaining spatial coherence. By explicitly guiding the reasoning process through structured relational representations, our approach improves both interpretability and control over spatial relationships. We evaluate our method on a new text-guided layout editing benchmark encompassing sorting, spatial alignment, and room-editing tasks. Our training paradigm yields an average 15% improvement in IoU and 25% reduction in center-distance error compared to Chain of Thought Fine-tuning (CoT-SFT) and vanilla GRPO baselines. Compared to SOTA zero-shot LLMs, our best models achieve up to 20% higher mIoU, demonstrating markedly improved spatial precision.
Paper Structure (31 sections, 4 equations, 9 figures, 5 tables)

This paper contains 31 sections, 4 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Preliminaries
  5. Scene graph representation.
  6. CoT-SFT and GRPO Training.
  7. Dataset Creation
  8. Sorting task.
  9. Spatial alignment task.
  10. Room editing task.
  11. 3D Layout Reinforcement Learning
  12. IoU reward.
  13. Collision reward.
  14. Format reward.
  15. Experiments
  16. ...and 16 more sections

Figures (9)

  • Figure 1: We introduce 3D-Layout-R1, which performs multi-step language-guided 3D layout editing, iteratively updating an initially randomized scene into a sequence of spatially consistent intermediate layouts.
  • Figure 2: Example from the synthetic 3D sorting benchmark. Given a instruction to group and sort objects by shape and height, our model generates a concise structured reasoning trace with JSON scene-graph updates that transforms the initial scene into the final layout, in contrast to a long, ambiguous free-form thinking path.
  • Figure 3: Overview of our training pipeline. The vision-language model predicts step-by-step layout edits from the instruction and initial scene graph, and rollouts are optimized using a combination of format, IoU, and collision-free rewards.
  • Figure 4: Example of text-guided 3D room layout reasoning, showing how the model interprets constraints to update object poses step by step and validate distances.
  • Figure 5: Out-of-domain warehouse simulation results showing our model correctly following user instructions with 3D boxes.
  • ...and 4 more figures