Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Large Language Models for Real-World Engineering Tasks

Rene Heesch, Sebastian Eilermann, Alexander Windmann, Alexander Diedrich, Philipp Rosenthal, Oliver Niggemann

TL;DR

This paper addresses the gap in engineering-specific evaluation of large language models by introducing a public dataset of over 100 real-world engineering questions and forecasting tasks derived from production systems. It benchmarks four LLMs (two local and two cloud-based) across five engineering competencies, including world-model generation, relational reasoning, design-intent inference, temporal-causal reasoning, and forecasting, using real plant and simulated environments. The results show LLMs excel at basic temporal/structural reasoning and maintaining consistent internal representations but struggle with non-local dependencies, abstract modeling, and accurate forecasting in complex, nonlinear CPS; GPT-4o often leads among models, while local LLMs are better as assistive tools. The study highlights the need for integrating formal domain models and careful oversight for reliable engineering deployment, and it outlines concrete guidelines and future directions for evaluation metrics and hybrid approaches.

Abstract

Large Language Models (LLMs) are transformative not only for daily activities but also for engineering tasks. However, current evaluations of LLMs in engineering exhibit two critical shortcomings: (i) the reliance on simplified use cases, often adapted from examination materials where correctness is easily verifiable, and (ii) the use of ad hoc scenarios that insufficiently capture critical engineering competencies. Consequently, the assessment of LLMs on complex, real-world engineering problems remains largely unexplored. This paper addresses this gap by introducing a curated database comprising over 100 questions derived from authentic, production-oriented engineering scenarios, systematically designed to cover core competencies such as product design, prognosis, and diagnosis. Using this dataset, we evaluate four state-of-the-art LLMs, including both cloud-based and locally hosted instances, to systematically investigate their performance on complex engineering tasks. Our results show that LLMs demonstrate strengths in basic temporal and structural reasoning but struggle significantly with abstract reasoning, formal modeling, and context-sensitive engineering logic.

Evaluating Large Language Models for Real-World Engineering Tasks

TL;DR

This paper addresses the gap in engineering-specific evaluation of large language models by introducing a public dataset of over 100 real-world engineering questions and forecasting tasks derived from production systems. It benchmarks four LLMs (two local and two cloud-based) across five engineering competencies, including world-model generation, relational reasoning, design-intent inference, temporal-causal reasoning, and forecasting, using real plant and simulated environments. The results show LLMs excel at basic temporal/structural reasoning and maintaining consistent internal representations but struggle with non-local dependencies, abstract modeling, and accurate forecasting in complex, nonlinear CPS; GPT-4o often leads among models, while local LLMs are better as assistive tools. The study highlights the need for integrating formal domain models and careful oversight for reliable engineering deployment, and it outlines concrete guidelines and future directions for evaluation metrics and hybrid approaches.

Abstract

Large Language Models (LLMs) are transformative not only for daily activities but also for engineering tasks. However, current evaluations of LLMs in engineering exhibit two critical shortcomings: (i) the reliance on simplified use cases, often adapted from examination materials where correctness is easily verifiable, and (ii) the use of ad hoc scenarios that insufficiently capture critical engineering competencies. Consequently, the assessment of LLMs on complex, real-world engineering problems remains largely unexplored. This paper addresses this gap by introducing a curated database comprising over 100 questions derived from authentic, production-oriented engineering scenarios, systematically designed to cover core competencies such as product design, prognosis, and diagnosis. Using this dataset, we evaluate four state-of-the-art LLMs, including both cloud-based and locally hosted instances, to systematically investigate their performance on complex engineering tasks. Our results show that LLMs demonstrate strengths in basic temporal and structural reasoning but struggle significantly with abstract reasoning, formal modeling, and context-sensitive engineering logic.
Paper Structure (10 sections, 1 figure, 5 tables)

This paper contains 10 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. State of the Art
  3. Analysis of Research Questions
  4. RQ1: Can LLMs generate world models?
  5. RQ2: Can LLMs accurately interpret and manage transitional and non-local relations?
  6. RQ3: How effectively can LLMs infer implicit design intentions underlying engineering models?
  7. RQ4: Are LLMs capable of temporal reasoning and capturing complex causal dependencies?
  8. RQ5: How accurately can LLMs predict system behavior?
  9. Discussion
  10. Conclusion

Figures (1)

  • Figure 1: Modular glass foaming plant that served as first use case. The productions sells Primer Application, Polyurethane Foaming, and Trimming & Final Inspection are highlighted.