GenCAD: Image-Conditioned Computer-Aided Design Generation with Transformer-Based Contrastive Representation and Diffusion Priors

TL;DR

GenCAD tackles the challenge of generating editable, manufacturable CAD shapes by treating CAD as a language with parametric command sequences and conditioning generation on images. It introduces a four-part framework—Command Sequence Reconstruction (CSR), Contrastive CAD–Image Pre-training (CCIP), CAD Diffusion Prior (CDP), and a CAD Decoder—that jointly learn a latent space for CAD commands and their visual counterparts, then synthesize complete CAD programs from image latents. Empirical results show superior CAD sequence reconstruction versus prior autoencoders, strong image-based CAD retrieval performance, and competitive unconditional and conditional CAD generation across standard metrics, highlighting the practical potential for image-to-CAD design workflows. While promising, the approach currently uses a limited CAD vocabulary and lacks guaranteed validity of generated CADs, suggesting future directions in richer tokens, CAD-verification feedback, and deployment with real-world, noisy imagery in commercial CAD environments.

Abstract

The creation of manufacturable and editable 3D shapes through Computer-Aided Design (CAD) remains a highly manual and time-consuming task, hampered by the complex topology of boundary representations of 3D solids and unintuitive design tools. While most work in the 3D shape generation literature focuses on representations like meshes, voxels, or point clouds, practical engineering applications demand the modifiability and manufacturability of CAD models and the ability for multi-modal conditional CAD model generation. This paper introduces GenCAD, a generative model that employs autoregressive transformers with a contrastive learning framework and latent diffusion models to transform image inputs into parametric CAD command sequences, resulting in editable 3D shape representations. Extensive evaluations demonstrate that GenCAD significantly outperforms existing state-of-the-art methods in terms of the unconditional and conditional generations of CAD models. Additionally, the contrastive learning framework of GenCAD facilitates the retrieval of CAD models using image queries from large CAD databases, which is a critical challenge within the CAD community. Our results provide a significant step forward in highlighting the potential of generative models to expedite the entire design-to-production pipeline and seamlessly integrate different design modalities.

Figures (15)

• Figure 1: GenCAD demonstrates generative CAD models conditioned on images or sketches, where the CAD model is generated sequentially using a language-like representation of CAD operations
• Figure 2: GenCAD: The proposed framework, GenCAD, consists of four steps: 1) a transformer-based encoder-decoder architecture is trained autoregressively to learn the latent representation of the vectorized CAD commands, 2) a contrastive-learning based model is used to learn the joint representation of the latent space of CAD command sequence and CAD image, 3) image conditional CAD generation can be achieved by sampling CAD latents from the diffusion model conditioned on image latents, and 4) using the trained transformer decoder to predict the CAD commands autoregressively. Note that denotes a frozen model that is not updated during training.
• Figure 3: CAD as a language modeling problem using geometry kernel: Our main idea is based on the fact that real-world CAD design is sequential and a learning-based approach should capture the correlation in the sequential design from a large-scale dataset. This sequential approach converts the CAD problem into a language modeling problem where vector representation of CAD commands, $\mathbf{c}_i \in \mathbb{R}^d$, are fed into an off-the-shelf geometry modeling kernel to sequentially create the 3D solid geometry such as Boundary-representation (B-rep). In this toy example, the sequence is as follows: line $\rightarrow$ line $\rightarrow$ line $\rightarrow$ arc $\rightarrow$ circle $\rightarrow$ extrusion.
• Figure 4: CAD Diffusion Prior (CDP): The diffusion prior takes CAD latent as well as optional image latent as input. The denoising model of the diffusion prior is based on MLP-ResNet architecture proposed by gorishniy2021revisiting.
• Figure 5: Performance of GenCAD-autoencoder corresponding to the length of CAD command sequences