On the Evaluation of Engineering Artificial General Intelligence

TL;DR

This work presents an evaluation framework for engineering artificial general intelligence (eAGI) grounded in a Bloom's taxonomy–based cognitive hierarchy, extended with domain-specific metadata to capture cross-domain engineering complexity. It combines reusable question templates with system-context, cognitive targets, and task types to generate diverse, domain-relevant benchmarks that span from factual recall to open-ended design synthesis and meta-reasoning. The framework is demonstrated on a propeller-motor matching problem for an eVTOL, illustrating how textual tasks and engineering artifacts (CAD/SysML) can be evaluated in a pluggable, automated fashion, with appropriate human-in-the-loop involvement at higher levels. This taxonomy-driven approach aims to enable principled, scalable benchmarking of eAGI agents that can meaningfully collaborate with human engineers across multiple engineering domains, addressing a critical need for robust evaluation in cognitive engineering automation.

Abstract

We discuss the challenges and propose a framework for evaluating engineering artificial general intelligence (eAGI) agents. We consider eAGI as a specialization of artificial general intelligence (AGI), deemed capable of addressing a broad range of problems in the engineering of physical systems and associated controllers. We exclude software engineering for a tractable scoping of eAGI and expect dedicated software engineering AI agents to address the software implementation challenges. Similar to human engineers, eAGI agents should possess a unique blend of background knowledge (recall and retrieve) of facts and methods, demonstrate familiarity with tools and processes, exhibit deep understanding of industrial components and well-known design families, and be able to engage in creative problem solving (analyze and synthesize), transferring ideas acquired in one context to another. Given this broad mandate, evaluating and qualifying the performance of eAGI agents is a challenge in itself and, arguably, a critical enabler to developing eAGI agents. In this paper, we address this challenge by proposing an extensible evaluation framework that specializes and grounds Bloom's taxonomy - a framework for evaluating human learning that has also been recently used for evaluating LLMs - in an engineering design context. Our proposed framework advances the state of the art in benchmarking and evaluation of AI agents in terms of the following: (a) developing a rich taxonomy of evaluation questions spanning from methodological knowledge to real-world design problems; (b) motivating a pluggable evaluation framework that can evaluate not only textual responses but also evaluate structured design artifacts such as CAD models and SysML models; and (c) outlining an automatable procedure to customize the evaluation benchmark to different engineering contexts.

Figures (2)

• Figure 1: The two key categories of tasks in engineering involve the synthesis of design from given requirements and the evaluation or analysis of a given design to measure its performance and compliance with constraints and regulations.
• Figure 2: 6 eAGI Levels based on Bloom's taxonomy