From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design

TL;DR

The paper evaluates vision-language models (VLMs) for engineering design across conceptual design, system/detail design, manufacturing/inspection, and engineering education tasks. By benchmarking GPT-4V and LLaVA 1.6 34B on over 1000 queries, it demonstrates that VLMs can support sketch understanding, description generation, and high-level design reasoning, but face notable gaps in precise geometry, CAD generation, topology optimization metrics, and numerical reasoning. The authors provide standardized datasets and protocols to enable future benchmarking, revealing that current VLMs offer valuable but task-limited assistance, requiring human oversight and integration with external tools for rigorous engineering workflows. The work highlights practical implications for deploying VLMs in engineering design, including potential gains in concept exploration and data cataloging, while outlining important limitations and directions for future research. Overall, the study lays groundwork for standardized evaluation of vision-language models in engineering and motivates continued development of multi-image reasoning, precise spatial understanding, and CAD-oriented capabilities.

Abstract

Engineering design is undergoing a transformative shift with the advent of AI, marking a new era in how we approach product, system, and service planning. Large language models have demonstrated impressive capabilities in enabling this shift. Yet, with text as their only input modality, they cannot leverage the large body of visual artifacts that engineers have used for centuries and are accustomed to. This gap is addressed with the release of multimodal vision-language models (VLMs), such as GPT-4V, enabling AI to impact many more types of tasks. Our work presents a comprehensive evaluation of VLMs across a spectrum of engineering design tasks, categorized into four main areas: Conceptual Design, System-Level and Detailed Design, Manufacturing and Inspection, and Engineering Education Tasks. Specifically in this paper, we assess the capabilities of two VLMs, GPT-4V and LLaVA 1.6 34B, in design tasks such as sketch similarity analysis, CAD generation, topology optimization, manufacturability assessment, and engineering textbook problems. Through this structured evaluation, we not only explore VLMs' proficiency in handling complex design challenges but also identify their limitations in complex engineering design applications. Our research establishes a foundation for future assessments of vision language models. It also contributes a set of benchmark testing datasets, with more than 1000 queries, for ongoing advancements and applications in this field.

Figures (29)

• Figure 1: We explored GPT-4V and LLaVA 1.6 34B's ability to perform numerous engineering design tasks that utilize both visual and textual information. Panel "Textbook Problems" adapted from OCW_2007 under CC BY-NC-SA 4.0. Panel "Material Inspection" adapted from mundt2019meta under its specific license.
• Figure 2: Ten conceptual designs of novel milk frothers. We task GPT-4V with assessing the similarity of these designs to one another. The handwritten descriptions at the bottom of each design are referred to as "text descriptions."
• Figure 3: Assess design similarity.
• Figure 4: A map of the milk frother design sketches where sketches that are closer to each other are more similar. These are based on the responses by GPT-4V for 360 triplet similarity queries. We observe that the map clusters similar designs together and places unique designs further away from other designs.
• Figure 5: Match a design to the correct description.