3D Space as a Scratchpad for Editable Text-to-Image Generation

TL;DR

The paper tackles the challenge that visual-language models struggle with precise spatial reasoning and compositional coherence. It introduces a 3D spatial scratchpad that converts prompts into an editable 3D scene, coordinated by an agent-driven pipeline for subject instantiation, 3D placement, orientation editing, and camera selection, followed by depth- and identity-conditioned image generation. The approach yields substantial gains on complex prompts (e.g., GenAI-Bench) and enables reliable 3D-aware editing, demonstrating improved text alignment while preserving subject identities and scene backgrounds. By grounding image synthesis in a 3D reasoning substrate, the work proposes a new paradigm for vision-language models that reason in space as well as language, enabling more controllable and faithful multi-subject generation.

Abstract

Recent progress in large language models (LLMs) has shown that reasoning improves when intermediate thoughts are externalized into explicit workspaces, such as chain-of-thought traces or tool-augmented reasoning. Yet, visual language models (VLMs) lack an analogous mechanism for spatial reasoning, limiting their ability to generate images that accurately reflect geometric relations, object identities, and compositional intent. We introduce the concept of a spatial scratchpad -- a 3D reasoning substrate that bridges linguistic intent and image synthesis. Given a text prompt, our framework parses subjects and background elements, instantiates them as editable 3D meshes, and employs agentic scene planning for placement, orientation, and viewpoint selection. The resulting 3D arrangement is rendered back into the image domain with identity-preserving cues, enabling the VLM to generate spatially consistent and visually coherent outputs. Unlike prior 2D layout-based methods, our approach supports intuitive 3D edits that propagate reliably into final images. Empirically, it achieves a 32% improvement in text alignment on GenAI-Bench, demonstrating the benefit of explicit 3D reasoning for precise, controllable image generation. Our results highlight a new paradigm for vision-language models that deliberate not only in language, but also in space. Code and visualizations at https://oindrilasaha.github.io/3DScratchpad/

Figures (12)

• Figure 1: Spatial scratchpads enable structured 3D reasoning for controllable image generation. Our method constructs a 3D spatial scratchpad from a text prompt, representing subjects as editable meshes with explicit geometry and spatial relations. Users or language agents can manipulate this scene --- moving, resizing, or reorienting objects --- through either text-based or interactive edits. The updated 3D configuration is re-rendered via a 3D-guided text-to-image pipeline, producing identity-preserving and spatially coherent images that remain faithful to the original prompt. This demonstrates how 3D reasoning serves as an effective intermediate workspace linking linguistic intent and precise visual synthesis.
• Figure 2: Overview of a 3D spatial scratchpad. Given an input prompt $P$ we illustrate how our method uses a 3D space as an underlying representation to generate an image that has superior alignment to the prompt. Agent ① is responsible for decomposing the input prompt into subjects and background. Agent ② provides 3D bounding boxes for each subject. We render the scratchpad and subsequently generate an image based on these placements which is then given to agent ③ that adjusts transformations of the meshes. Finally, agent ④ chooses the best camera viewpoint from a set of proposals to generate the final image.
• Figure 3: Identity preservation improves prompt adherence. We show that complex planning among multiple subjects even when guided with only depth and prompt can lead to loss of text alignment. In contrast, we opt to use depth, prompt, and identity to generate the images thus preserving prompt adherence.
• Figure 4: Progression of 3D spatial scratchpad. We show examples where the planning conducted by agents ③ and ④ are crucial for being faithful to the text. In the first example, only ③ is useful, while on the other both ③ and ④ are required.
• Figure 5: Qualitative comparison on GenAI-Bench and CompoundPrompts datasets. We show examples where our method correctly captures the prompt, while the baseline methods fail. The GenAI-Bench dataset examples capture position, action, negation, and attribute planning. CompoundPrompts dataset examples show accurate attribute planning combined with spatial planning for many subjects.