Generating CAD Code with Vision-Language Models for 3D Designs

TL;DR

The paper tackles generating manufacturable CAD designs from natural language prompts, where initial CAD code often fails to satisfy specifications. It presents CADCodeVerify, a vision-language model–driven refinement loop that generates verification questions and ameliorative feedback to iteratively improve CAD code, with a three-stage workflow: Code Generation, Code Execution, and Code Refinement; refinement is performed for up to $N$ iterations and may use reference images $I_{ref}$ and feedback $F_{ref}$. A new CADPrompt benchmark with 200 prompts and ground-truth code enables cross-model evaluation across GPT-4, Gemini, and CodeLlama. Results show CADCodeVerify improves 3D object quality and compile success, reducing $D(P,Q)$ by 7.3% and increasing the success rate by 5.0% on GPT-4, demonstrating a model-agnostic feedback loop for manufacturable CAD design.

Abstract

Generative AI has transformed the fields of Design and Manufacturing by providing efficient and automated methods for generating and modifying 3D objects. One approach involves using Large Language Models (LLMs) to generate Computer- Aided Design (CAD) scripting code, which can then be executed to render a 3D object; however, the resulting 3D object may not meet the specified requirements. Testing the correctness of CAD generated code is challenging due to the complexity and structure of 3D objects (e.g., shapes, surfaces, and dimensions) that are not feasible in code. In this paper, we introduce CADCodeVerify, a novel approach to iteratively verify and improve 3D objects generated from CAD code. Our approach works by producing ameliorative feedback by prompting a Vision-Language Model (VLM) to generate and answer a set of validation questions to verify the generated object and prompt the VLM to correct deviations. To evaluate CADCodeVerify, we introduce, CADPrompt, the first benchmark for CAD code generation, consisting of 200 natural language prompts paired with expert-annotated scripting code for 3D objects to benchmark progress. Our findings show that CADCodeVerify improves VLM performance by providing visual feedback, enhancing the structure of the 3D objects, and increasing the success rate of the compiled program. When applied to GPT-4, CADCodeVerify achieved a 7.30% reduction in Point Cloud distance and a 5.0% improvement in success rate compared to prior work

Figures (18)

• Figure 1: Our approach enables VLMs to automatically generate and refine 3D objects through a CAD scripting code (e.g., CADQuery) in three steps; (1) Code Generation, where the VLM generates CAD scripting code from a language prompt, (2) Code Execution, where the code generated by the model is rendered as a 3D object through a compiler, and (3) Code Refinement, wherein the language model engages in a self-initiated question-answering process to validate the generated object, with respect to the initial prompt, to generate actionable feedback to refine the code.
• Figure 2: Examples of 3D objects generated by GPT4 after utilizing each feedback method for code refinement from our CADPrompt dataset.
• Figure 3: Examples of 3D objects from the CADPrompt dataset.
• Figure 4: Compile rate comparison across different feedback mechanisms, categorized by difficulty and mesh complexity. Each figure corresponds to a specific subset of the CADPrompt dataset, covering both generated and refined objects. The plots for the “Hard” and “Complex” splits indicate that the feedback approach from 3D-Premise reduces GPT-4's compile rate in object generation.
• Figure 5: An example of a subset of questions and answers generated by CADCodeVerify.