ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling

TL;DR

ShapeCraft addresses the challenge of producing production-ready, editable 3D assets from natural language by introducing a Graph-based Procedural Shape (GPS) representation and a multi-agent pipeline. The Parser, Coder, and Evaluator collaboratively construct and refine a GPS, whose leaves become geometry programs executed in Blender, with a component-aware BRDF painting stage guided by Score Distillation Sampling. Key contributions include a depth-1 GPS graph with per-node attributes, iterative multi-path modeling to explore alternatives, and a CASD texture framework that aligns material properties with prompts. The approach yields more structurally sound geometries, richer semantic detail, and interactive post-modeling capabilities, offering practical benefits for artists and AI-enabled design workflows, while acknowledging limitations in handling highly organic forms and dependency on prompt quality.

Abstract

3D generation from natural language offers significant potential to reduce expert manual modeling efforts and enhance accessibility to 3D assets. However, existing methods often yield unstructured meshes and exhibit poor interactivity, making them impractical for artistic workflows. To address these limitations, we represent 3D assets as shape programs and introduce ShapeCraft, a novel multi-agent framework for text-to-3D generation. At its core, we propose a Graph-based Procedural Shape (GPS) representation that decomposes complex natural language into a structured graph of sub-tasks, thereby facilitating accurate LLM comprehension and interpretation of spatial relationships and semantic shape details. Specifically, LLM agents hierarchically parse user input to initialize GPS, then iteratively refine procedural modeling and painting to produce structured, textured, and interactive 3D assets. Qualitative and quantitative experiments demonstrate ShapeCraft's superior performance in generating geometrically accurate and semantically rich 3D assets compared to existing LLM-based agents. We further show the versatility of ShapeCraft through examples of animated and user-customized editing, highlighting its potential for broader interactive applications.

Figures (16)

• Figure 1: Qualitative results of ShapeCraft. Our agentic text-to-shape framework generates bounding volumes, raw meshes and textured shapes, enabling advanced post-modeling interactions like shape editing and animation tasks. Project page is https://sanbingyouyong.github.io/shapecraft/.
• Figure 2: Overview of ShapeCraft system, a multi-agent system to produce structured and post-modeling friendly 3d assets. Given a shape description $x$, the Parser agent hierarchically decomposes the shape and initializes Graph-based Procedural Shape representation $\mathcal{G}$. Then, each node $v_i$ is iteratively modeled by updating its code snippet using a multi-path strategy, with reinforcement from the Coder and Evaluator agents. Finally, a component-aware score distillation learns a texture field from the resulting mesh to produce textured results.
• Figure 3: Qualitative comparison with optimization-based method. ShapeCraft consistently produces more structured meshes with better prompt following in both geometry and texture (e.g. "rust and dirt spots" in air conditioner). Red and blue areas highlight specific zoom-in observations.
• Figure 4: Qualitative comparison of raw mesh against LLM-based methods. ShapeCraft demonstrates superior performance for both intricate ("Banjo") and simpler ("bookshelf") cases. The black highlighted areas reveal ShapeCraft's capability to generate complex shape details, benefiting from component decomposition in GPS representation.
• Figure 5: Showcases of iterative refinement for multi-path sampling. (a) Corrects z‐fighting artifacts on extraneous buttons. (b) Eliminates redundant bevel operations. (c) Shows a trajectory degraded by a poor initial sample, while (d) demonstrates an alternative path that still yields acceptable geometry.