LLM Evaluation Based on Aerospace Manufacturing Expertise: Automated Generation and Multi-Model Question Answering

TL;DR

This work tackles the risk of LLM hallucinations in aerospace manufacturing by proposing a domain-specific evaluation framework built on a knowledge-grounded MCQ bank derived from authoritative texts. It employs automated MCQ generation, expert validation, and a custom scoring scheme to assess multiple LLMs across three sub-domains, revealing substantial variability and notable challenges in materials knowledge. The study demonstrates that while some models approach modest accuracy (~50%), domain-specific knowledge remains a critical bottleneck, underscoring the need for grounding, verification, and possibly retrieval-augmented approaches for safe deployment. The framework provides both a theoretical foundation and practical guidance for responsible integration of LLMs into aerospace manufacturing tasks.

Abstract

Aerospace manufacturing demands exceptionally high precision in technical parameters. The remarkable performance of Large Language Models (LLMs), such as GPT-4 and QWen, in Natural Language Processing has sparked industry interest in their application to tasks including process design, material selection, and tool information retrieval. However, LLMs are prone to generating "hallucinations" in specialized domains, producing inaccurate or false information that poses significant risks to the quality of aerospace products and flight safety. This paper introduces a set of evaluation metrics tailored for LLMs in aerospace manufacturing, aiming to assess their accuracy by analyzing their performance in answering questions grounded in professional knowledge. Firstly, key information is extracted through in-depth textual analysis of classic aerospace manufacturing textbooks and guidelines. Subsequently, utilizing LLM generation techniques, we meticulously construct multiple-choice questions with multiple correct answers of varying difficulty. Following this, different LLM models are employed to answer these questions, and their accuracy is recorded. Experimental results demonstrate that the capabilities of LLMs in aerospace professional knowledge are in urgent need of improvement. This study provides a theoretical foundation and practical guidance for the application of LLMs in aerospace manufacturing, addressing a critical gap in the field.

Figures (2)

• Figure 1: Knowledge Distortion Pathway: From LLM hallucination to aviation risk (Data sources: HB2023, NASA2021)
• Figure 2: Evaluation Methodology