Don't Mesh with Me: Generating Constructive Solid Geometry Instead of Meshes by Fine-Tuning a Code-Generation LLM

TL;DR

This work targets precise, parametric CAD-style 3D geometry by generating surface-based Constructive Solid Geometry (CSG) rather than meshes. It introduces a pipeline that converts Boundary Representation (BREP) data into CSG Python scripts, augments with natural-language annotations via GPT-4, and fine-tunes a code-generation LLM to complete 3D geometry and follow text instructions for simple parts. The approach claims to be the first deep-learning model specialized for CSG generation and the first LLM fine-tuned for 3D geometry completion guided by language, with STEP-format compatibility across CAD tools. Experimental results on a curated ABC dataset show the model learns plausible, syntactically valid CSG constructions and can be steered by text for basic geometries, illustrating a path toward more controllable CAD automation.

Abstract

While recent advancements in machine learning, such as LLMs, are revolutionizing software development and creative industries, they have had minimal impact on engineers designing mechanical parts, which remains largely a manual process. Existing approaches to generating 3D geometry most commonly use meshes as a 3D representation. While meshes are suitable for assets in video games or animations, they lack sufficient precision and adaptability for mechanical engineering purposes. This paper introduces a novel approach for the generation of 3D geometry that generates surface-based Constructive Solid Geometry (CSG) by leveraging a code-generation LLM. First, we create a dataset of 3D mechanical parts represented as code scripts by converting Boundary Representation geometry (BREP) into CSG-based Python scripts. Second, we create annotations in natural language using GPT-4. The resulting dataset is used to fine-tune a code-generation LLM. The fine-tuned LLM can complete geometries based on positional input and natural language in a plausible way, demonstrating geometric understanding.

Figures (10)

• Figure 1: Point Cloud
• Figure 2: Voxel Grid
• Figure 3: Mesh
• Figure 4: CAD/CSG
• Figure 6: Data generation pipeline. The 3D geometry is first converted into a Graph and then sequentialized. The sequence is split into in- and output and converted into a python script. In parallel the output is rendered as a 2D image and converted into a text annotation which is finally added to the script data.