EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark

TL;DR

EEE-Bench introduces a pioneering multimodal benchmark for electrical and electronics engineering, targeting the evaluation of LLMs and LMMs on visually rich, real-world engineering problems. It comprises 2,860 questions across 10 subdomains and includes diverse visual contexts, with a rigorous data collection, curation, and release pipeline to enable reproducible research. A comprehensive evaluation of 17 models (open- and closed-source) reveals pronounced gaps in current models, with average accuracies ranging from 19.48% to 46.78% and a notable laziness phenomenon where models rely on text over visual cues. The work demonstrates the critical role of vision in EEE problem solving, provides in-depth error analysis, and offers a resource to guide future improvements in multimodal reasoning for engineering tasks with significant real-world impact.

Abstract

Recent studies on large language models (LLMs) and large multimodal models (LMMs) have demonstrated promising skills in various domains including science and mathematics. However, their capability in more challenging and real-world related scenarios like engineering has not been systematically studied. To bridge this gap, we propose EEE-Bench, a multimodal benchmark aimed at assessing LMMs' capabilities in solving practical engineering tasks, using electrical and electronics engineering (EEE) as the testbed. Our benchmark consists of 2860 carefully curated problems spanning 10 essential subdomains such as analog circuits, control systems, etc. Compared to benchmarks in other domains, engineering problems are intrinsically 1) more visually complex and versatile and 2) less deterministic in solutions. Successful solutions to these problems often demand more-than-usual rigorous integration of visual and textual information as models need to understand intricate images like abstract circuits and system diagrams while taking professional instructions, making them excellent candidates for LMM evaluations. Alongside EEE-Bench, we provide extensive quantitative evaluations and fine-grained analysis of 17 widely-used open and closed-sourced LLMs and LMMs. Our results demonstrate notable deficiencies of current foundation models in EEE, with an average performance ranging from 19.48% to 46.78%. Finally, we reveal and explore a critical shortcoming in LMMs which we term laziness: the tendency to take shortcuts by relying on the text while overlooking the visual context when reasoning for technical image problems. In summary, we believe EEE-Bench not only reveals some noteworthy limitations of LMMs but also provides a valuable resource for advancing research on their application in practical engineering tasks, driving future improvements in their capability to handle complex, real-world scenarios.

Figures (42)

• Figure 1: Comparison between EEE-Bench and MathVista problems: EEE problems feature significantly more complex and intricate visual information compared to mathematical problems, requiring specific technical knowledge to interpret diagrams effectively.
• Figure 1: Key Statistics of EEE-Bench.
• Figure 2: Overview of performance of left: open source models and right: close source models.
• Figure 3: Subject Distribution of EEE-Bench. DLC&MPD: Digital Logic Circuits and Microprocessor Design, PS: power systems. NA: network analysis
• Figure 4: Comparison between solutions of logical circuits and circuit theory. Logic Circuits demand a thorough comprehension of logical structuring and timing analysis, yet they do not heavily rely on computation. In contrast, problems of circuit theory involve solving complex equations and applying numerical methods to analyze circuits.