Proc3D: Procedural 3D Generation and Parametric Editing of 3D Shapes with Large Language Models

TL;DR

Proc3D introduces Procedural Compact Graph ($PCG$), a compact, human-readable graph that encodes high-level 3D operations and exposes per-node parameters for true real-time edits. An LLM constructs and updates $PCG$s from natural language prompts, while an interpreter compiles them into engine-specific code (e.g., Blender/Unity), enabling seamless generation and live parametric edits. By extracting $PCG$ from PartNet and training an LLaMA-3 model on 63K instruction-graph pairs, Proc3D achieves strong text-to-3D alignment and highly efficient editing, outperforming prior methods in both representation compactness and ULIP scores. This approach enables rapid, text-driven 3D design with precise control, potentially transforming interactive design pipelines that require iterative, label-conditioned modifications in real time.

Abstract

Generating 3D models has traditionally been a complex task requiring specialized expertise. While recent advances in generative AI have sought to automate this process, existing methods produce non-editable representation, such as meshes or point clouds, limiting their adaptability for iterative design. In this paper, we introduce Proc3D, a system designed to generate editable 3D models while enabling real-time modifications. At its core, Proc3D introduces procedural compact graph (PCG), a graph representation of 3D models, that encodes the algorithmic rules and structures necessary for generating the model. This representation exposes key parameters, allowing intuitive manual adjustments via sliders and checkboxes, as well as real-time, automated modifications through natural language prompts using Large Language Models (LLMs). We demonstrate Proc3D's capabilities using two generative approaches: GPT-4o with in-context learning (ICL) and a fine-tuned LLAMA-3 model. Experimental results show that Proc3D outperforms existing methods in editing efficiency, achieving more than 400x speedup over conventional approaches that require full regeneration for each modification. Additionally, Proc3D improves ULIP scores by 28%, a metric that evaluates the alignment between generated 3D models and text prompts. By enabling text-aligned 3D model generation along with precise, real-time parametric edits, Proc3D facilitates highly accurate text-based image editing applications.

Figures (17)

• Figure 1: 3D Generation and Editing Pipeline. Given a text prompt, a Large Language Model (LLM) constructs the Procedural Compact Graph (PCG), which is interpreted into software-specific formats suitable for 3D modeling engine such as Blender or Unity3D. Subsequent edits submitted via text prompt are integrated by the LLM to update the graph and, consequently, the 3D mesh.
• Figure 2: An example of the procedural compact graph (PCG). Each line is a node in the procedural 3D graph that defines the operation performed on the connected node. The graph is interpreted into blender's geometric node where the mesh can be visualized.
• Figure 3: Data Collection Pipeline. Given a dataset of the hierarchical part meshesmo2019partnetmo2019structurenet, we extract the PCG representations, and rendered multi-view images of the 3D model. Then, using LLaVA, we collect detail caption of the rendered views, which is passed through LLaMA-3 70B for the generation of corresponding instructions.
• Figure 4: Comparative Analysis of 3D Object Generation and Editing. (a) 3D Generation: Comparison of different methods for generating 3D models from text prompts, including SDFusion cheng2023sdfusion, LLaMA-Mesh llamamesh, LGM tang2024lgm, and our proposed method (Ours). Our method achieves more consistent, structured outputs across various prompts. (b) Text-Based 3D Editing: Our system introduces representation that enabled targeted edits on the generated 3D objects from text prompts. Other methods do not support editing the generated 3D models.
• Figure 5: Generation out of training classes. With training data limited to chairs, tables, beds, and storage, Proc3D can generate complex 3D categories that are not in the training examples.