Freehand Sketch Generation from Mechanical Components

TL;DR

A two-stage generative framework mimicking the human sketching behavior pattern, called MSFormer, is designed, which is the first time to produce humanoid freehand sketches tailored for mechanical components.

Abstract

Drawing freehand sketches of mechanical components on multimedia devices for AI-based engineering modeling has become a new trend. However, its development is being impeded because existing works cannot produce suitable sketches for data-driven research. These works either generate sketches lacking a freehand style or utilize generative models not originally designed for this task resulting in poor effectiveness. To address this issue, we design a two-stage generative framework mimicking the human sketching behavior pattern, called MSFormer, which is the first time to produce humanoid freehand sketches tailored for mechanical components. The first stage employs Open CASCADE technology to obtain multi-view contour sketches from mechanical components, filtering perturbing signals for the ensuing generation process. Meanwhile, we design a view selector to simulate viewpoint selection tasks during human sketching for picking out information-rich sketches. The second stage translates contour sketches into freehand sketches by a transformer-based generator. To retain essential modeling features as much as possible and rationalize stroke distribution, we introduce a novel edge-constraint stroke initialization. Furthermore, we utilize a CLIP vision encoder and a new loss function incorporating the Hausdorff distance to enhance the generalizability and robustness of the model. Extensive experiments demonstrate that our approach achieves state-of-the-art performance for generating freehand sketches in the mechanical domain. Project page: https://mcfreeskegen.github.io .

Figures (14)

• Figure 1: An overview of our method. (1) Stage-One: we generate contour sketches based on 26 viewpoints (represented by colorful points) of a cube (grey) . After that, Preprocessing and View Selection export appropriate contour sketches. (2) Stage-Two: By receiving initial strokes and features captured by our encoder from regular contour sketch, the stroke generator produces a set of strokes, which are next fed to a differentiable rasterizer to create a vector freehand sketch.
• Figure 2: Edge-constraint Initialization. (a) and (b) are results of segmenting through hole and overall segmentation of flange by SAM kirillov2023segment (distinguishing features through different coloring). (c) The saliency map generated from CLIP ViT activations. (d) and (e) are initial stroke locations (in red) in final distribution map and input. It is evident that our method accurately places initial strokes at features.
• Figure 3: Comparison to other methods for generating sketches of mechanical components.
• Figure 4: Comparison to other state-of-the-art method for generating sketches with a freehand style.
• Figure 5: Robust performance across abundant categories.