Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the effectiveness of Large Language Models in the mechanical design domain

Daniele Grandi, Fabian Riquelme

TL;DR

This work probes the effectiveness of large language models in the mechanical design domain by leveraging domain-specific semantic names from the ABC CAD dataset. It introduces two unsupervised evaluation tasks—binary sentence-pair entailment and zero-shot assembly-name prediction—and demonstrates that a contrastively pre-trained Bert variant substantially outperforms baselines on assembly-name prediction, achieving a top-1 accuracy of 0.386. The results reveal notable failure modes tied to data scarcity, high lexical diversity, and non-semantic labels, while offering concrete guidance on fine-tuning strategies (learning rate, sequence length, dropout, and multi-head attention) to mitigate overfitting. Overall, the study provides a foundation for applying domain-adapted transformer models to CAD-related language and suggests directions for generative and data-augmentation approaches to better support mechanical designers.

Abstract

In this work, we seek to understand the performance of large language models in the mechanical engineering domain. We leverage the semantic data found in the ABC dataset, specifically the assembly names that designers assigned to the overall assemblies, and the individual semantic part names that were assigned to each part. After pre-processing the data we developed two unsupervised tasks to evaluate how different model architectures perform on domain-specific data: a binary sentence-pair classification task and a zero-shot classification task. We achieved a 0.62 accuracy for the binary sentence-pair classification task with a fine-tuned model that focuses on fighting over-fitting: 1) modifying learning rates, 2) dropout values, 3) Sequence Length, and 4) adding a multi-head attention layer. Our model on the zero-shot classification task outperforms the baselines by a wide margin, and achieves a top-1 classification accuracy of 0.386. The results shed some light on the specific failure modes that arise when learning from language in this domain.

On the effectiveness of Large Language Models in the mechanical design domain

TL;DR

This work probes the effectiveness of large language models in the mechanical design domain by leveraging domain-specific semantic names from the ABC CAD dataset. It introduces two unsupervised evaluation tasks—binary sentence-pair entailment and zero-shot assembly-name prediction—and demonstrates that a contrastively pre-trained Bert variant substantially outperforms baselines on assembly-name prediction, achieving a top-1 accuracy of 0.386. The results reveal notable failure modes tied to data scarcity, high lexical diversity, and non-semantic labels, while offering concrete guidance on fine-tuning strategies (learning rate, sequence length, dropout, and multi-head attention) to mitigate overfitting. Overall, the study provides a foundation for applying domain-adapted transformer models to CAD-related language and suggests directions for generative and data-augmentation approaches to better support mechanical designers.

Abstract

In this work, we seek to understand the performance of large language models in the mechanical engineering domain. We leverage the semantic data found in the ABC dataset, specifically the assembly names that designers assigned to the overall assemblies, and the individual semantic part names that were assigned to each part. After pre-processing the data we developed two unsupervised tasks to evaluate how different model architectures perform on domain-specific data: a binary sentence-pair classification task and a zero-shot classification task. We achieved a 0.62 accuracy for the binary sentence-pair classification task with a fine-tuned model that focuses on fighting over-fitting: 1) modifying learning rates, 2) dropout values, 3) Sequence Length, and 4) adding a multi-head attention layer. Our model on the zero-shot classification task outperforms the baselines by a wide margin, and achieves a top-1 classification accuracy of 0.386. The results shed some light on the specific failure modes that arise when learning from language in this domain.
Paper Structure (27 sections, 3 figures, 3 tables)

This paper contains 27 sections, 3 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND
  3. Usage of natural language data in design research
  4. Applying NLP to new domains
  5. Transformer Architecture
  6. METHODS
  7. Data Preprocessing
  8. Understanding if elements are part of an assembly
  9. Model Architecture
  10. Input Layer
  11. Base Model and Fine Tuning
  12. Evaluation
  13. Assembly name prediction
  14. Model architecture
  15. Evaluation
  16. ...and 12 more sections

Figures (3)

  • Figure 1: An example CAD assembly with each part's semantic name labeled.
  • Figure 2: Summary of the assembly name prediction task. First, (1) a text encoder is contrastively pre-trained on part name and assembly name pairs. Then, (2) batches of 100 unseen part name and assembly name sentences are encoded using the same text encoder, and (3) the dot product is computed between pairs. The assembly name with the highest value is chosen as the prediction.
  • Figure 3: Cosine similarity between the normalized embeddings of the first batch of 100 assembly names vs. 100 part names for (a) Bert base, (b) Bert fine-tuned, and (c) Bert pretrained. A 100% correct result would yield a figure where the items on the diagonal have the highest cosine similarity, indicating that the embedding of the assembly name and the embedding of the part names are most similar.