Spatial Chain-of-Thought: Bridging Understanding and Generation Models for Spatial Reasoning Generation

TL;DR

Spatial Chain-of-Thought (SCoT) tackles the challenge of embedding complex spatial reasoning into diffusion-based image generation. It introduces a grounded, two-component framework: SAGen, a spatially-aware diffusion backbone trained on dense caption-box data, and an off-the-shelf MLLM planner that outputs executable layouts via an interleaved text–coordinate representation. The authors also release dense grounding data (SCoT-DenseBox) and a high-aesthetic subset (SCoT-AestheticSFT) to train and fine-tune the system, achieving state-of-the-art results on T2ICoReBench, GenEval, OneIG-Bench, and COCO-MIG, and extending to image editing with IVEdit. The approach is plug-and-play and avoids joint pretraining while delivering precise spatial control over multi-object scenes.

Abstract

While diffusion models have shown exceptional capabilities in aesthetic image synthesis, they often struggle with complex spatial understanding and reasoning. Existing approaches resort to Multimodal Large Language Models (MLLMs) to enhance this capability. However, they either incur high computational costs through joint training or suffer from spatial information loss when relying solely on textual prompts. To alleviate these limitations, we propose a Spatial Chain-of-Thought (SCoT) framework, a plug-and-play approach that effectively bridges the reasoning capabilities of MLLMs with the generative power of diffusion models. Specifically, we first enhance the diffusion model's layout awareness by training it on an interleaved text-coordinate instruction format. We then leverage state-of-the-art MLLMs as planners to generate comprehensive layout plans, transferring their spatial planning capabilities directly to the generation process. Extensive experiments demonstrate that our method achieves state-of-the-art performance on image generation benchmarks and significantly outperforms baselines on complex reasoning tasks, while also showing strong efficacy in image editing scenarios.

Figures (4)

• Figure 1: Comparison of conditioning interfaces for spatially constrained text-to-image generation. (a) Continuous Bridge: MLLM encodes the prompt into continuous features and feeds them to the generation model. (b) Textual Bridge: MLLM expands the prompt into textual CoT, but spatial layout is compressed into language, and fine-grained structure is easily lost. (c) Spatial CoT Bridge (ours): MLLM outputs a Spatial CoT with object-level bounding boxes, which are rendered to an image via interleaved text--coordinates into a spatially-aware diffusion model, enabling faithful synthesis under strict spatial rules (e.g., a 3$\times$4 desk grid with required empty neighbors).
• Figure 2: Overview of our plug-and-play framework: an off-the-shelf MLLM planner converts a user prompt into a Spatial Chain-of-Thought layout plan in interleaved text–bounding-box tokens, which conditions a spatially-aware diffusion backbone to generate images with accurate multi-object layouts, counts, and spatial relations.
• Figure 3: Comparison of qualitative visualization in complex spatial scenes across Flux 1 Dev, Qwen-Image, Nano Banana Pro, and ours.
• Figure 4: Additional comparison of visualization in complex spatial scenes across Flux 1 Dev, Qwen-Image, Nano Banana Pro, and ours.