Shape of Thought: Progressive Object Assembly via Visual Chain-of-Thought

TL;DR

Shape-of-Thought presents a visual chain-of-thought framework that reframes shape generation as progressive object assembly, driven by interleaved textual rationales and rendered intermediate states. A unified multimodal Transformer learns from SoT-26K, a large dataset of stepwise, ground-truth assembly traces derived from PartNet CAD assets, and is evaluated with the T2S-CompBench benchmark that jointly measures final structure and trace faithfulness. Empirical results show SoT improves component numeracy and structural topology by substantial margins over direct-generation and text-only baselines, while providing explicit intermediate states that stabilize the generation trajectory. This process-supervised approach yields more transparent and controllable compositional generation, with code availability and dataset release planned, and sets a foundation for bridging 2D grounding with 3D lifting and downstream geometric reasoning.

Abstract

Multimodal models for text-to-image generation have achieved strong visual fidelity, yet they remain brittle under compositional structural constraints-notably generative numeracy, attribute binding, and part-level relations. To address these challenges, we propose Shape-of-Thought (SoT), a visual CoT framework that enables progressive shape assembly via coherent 2D projections without external engines at inference time. SoT trains a unified multimodal autoregressive model to generate interleaved textual plans and rendered intermediate states, helping the model capture shape-assembly logic without producing explicit geometric representations. To support this paradigm, we introduce SoT-26K, a large-scale dataset of grounded assembly traces derived from part-based CAD hierarchies, and T2S-CompBench, a benchmark for evaluating structural integrity and trace faithfulness. Fine-tuning on SoT-26K achieves 88.4% on component numeracy and 84.8% on structural topology, outperforming text-only baselines by around 20%. SoT establishes a new paradigm for transparent, process-supervised compositional generation. The code is available at https://anonymous.4open.science/r/16FE/. The SoT-26K dataset will be released upon acceptance.

Figures (21)

• Figure 1: Comparison of generation paradigms. While direct generation fails to capture details and text-based CoT leads to semantic binding failures, SoT improves structural compliance on our tasks. By decomposing complex prompts into sequential visual sub-goals, SoT corrects structural deficiencies and ensures the final output aligns with the text description.
• Figure 2: Overview of the SoT Ecosystem.Top: Statistics of the proposed SoT-26K dataset, showing the distribution of object categories and assembly step lengths. The data covers a wide range of structural complexities, from simple short-horizon objects to complex long-horizon assemblies. Bottom: The inference workflow of the SoT framework. Conditioned on a global goal prompt, the model autoregressively generates an interleaved multimodal trace, alternating between textual rationales and rendered intermediate states.
• Figure 3: Overview of the SoT framework. The architecture progressively evolves from Initial Assembly to Structural Evolution and Completion. At each step $n$, the model utilizes a Conceptual Layer where textual rationales ($z_n$) serve as a scaffold to guide the generation of grounded visual tokens ($v_n$). The bottom panel illustrates the tokenization implementation, where interleaved text and vision tokens are processed by a unified multimodal foundation model.
• Figure 4: The SoT-26K Construction Pipeline. We transform PartNet assets into multimodal traces through four stages: (A) Data Curation to filter and validate raw hierarchies; (B) Hierarchy Decomposition to enforce fine-grained assembly schedules; (C) Automated Rendering to generate cumulative intermediate states in Blender; and (D) Multimodal Annotation, where GPT-4o synthesizes step-wise rationales and goal prompts grounded in the visual evidence.
• Figure 5: Visualization of Visual Chain-of-Thought Traces. SoT generates objects by progressively decomposing the goal into sequential sub-tasks. At each step, the model generates a structural rationale followed by the corresponding grounded visual state.