CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention

TL;DR

CAD-SIGNet presents an end-to-end auto-regressive framework that infers a CAD design history from a point cloud by jointly learning CAD language and visual representations through layer-wise cross-attention. A novel Sketch Instance Guided Attention module concentrates cross-attention on sketch regions, enabling fine-grained sketch parameterization and interactive, multiple design options. The approach demonstrates strong improvements over prior methods in design history recovery and conditional auto-completion across multiple datasets and real-world scans, while maintaining a compact model size. These results advance interactive reverse engineering by producing editable, parametric CAD sequences from scans and enabling designer-guided exploration of design alternatives.

Abstract

Reverse engineering in the realm of Computer-Aided Design (CAD) has been a longstanding aspiration, though not yet entirely realized. Its primary aim is to uncover the CAD process behind a physical object given its 3D scan. We propose CAD-SIGNet, an end-to-end trainable and auto-regressive architecture to recover the design history of a CAD model represented as a sequence of sketch-and-extrusion from an input point cloud. Our model learns visual-language representations by layer-wise cross-attention between point cloud and CAD language embedding. In particular, a new Sketch instance Guided Attention (SGA) module is proposed in order to reconstruct the fine-grained details of the sketches. Thanks to its auto-regressive nature, CAD-SIGNet not only reconstructs a unique full design history of the corresponding CAD model given an input point cloud but also provides multiple plausible design choices. This allows for an interactive reverse engineering scenario by providing designers with multiple next-step choices along with the design process. Extensive experiments on publicly available CAD datasets showcase the effectiveness of our approach against existing baseline models in two settings, namely, full design history recovery and conditional auto-completion from point clouds.

Figures (13)

• Figure 1: Full design history recovery from an input point cloud (top-left) and CAD-SIGNet - user interaction (bottom-left and right).
• Figure 2: Method Overview. CAD-SIGNet (left) is composed of $\mathbf B$ Multi-Modal Transformer blocks, each consisting of an $\operatorname{LFA}$randlanet module to extract point features, $\mathbf F_{b}^v$, and a MSA transformer module for token features, $\mathbf F_{b}^c$. A SGA module (top right) combines $\mathbf F_{b}^v$ and $\mathbf F_{b}^c$ for CAD visual-language learning. A sketch instance (bottom right), $\mathbf I$, obtained from the predicted extrusion tokens is used to apply a mask, $\mathbf M_{\text{sga}}$ during CA to predict sketch tokens.
• Figure 3: Illustration of sketch and extrusion representations.
• Figure 4: Visual results of reconstruction from the CAD sequences predicted from input point clouds. Both DeepCAD Wu_2021_ICCV and CAD-SIGNet are trained on DeepCAD dataset Wu_2021_ICCV. Left: Results on DeepCAD dataset Wu_2021_ICCV. Middle: Cross-dataset results on CC3D dataset cc3d, Right: Cross-dataset results on Fusion360 dataset willis2021fusion.
• Figure 5: Visual results for auto-completion from user input on DeepCAD Wu_2021_ICCV dataset. From left to right, input point cloud, CAD model reconstruction from user input CAD sequence, SkexGen xu2022skexgen, HNC hnc, CAD-SIGNet (ours), and ground-truth.

Theorems & Definitions (1)

• Definition 1