FEABench: Evaluating Language Models on Multiphysics Reasoning Ability

TL;DR

FEABench tackles the challenge of using large language models to perform engineering-scale numerical simulations by enabling end-to-end reasoning and execution through the COMSOL Multiphysics API. It introduces FEABench Gold and FEABench Large datasets, plus an agent framework that grounds LLM reasoning with API feedback and iterative correction. The results show that grounding and multi-turn interaction improve executable code generation and problem solving, with Claude-3.5 Sonnet often leading among baselines and the Multi-Turn Agent achieving the best model-spec performance. The benchmark demonstrates a path toward autonomous, numerically grounded AI assistants for engineering design and analysis.

Abstract

Building precise simulations of the real world and invoking numerical solvers to answer quantitative problems is an essential requirement in engineering and science. We present FEABench, a benchmark to evaluate the ability of large language models (LLMs) and LLM agents to simulate and solve physics, mathematics and engineering problems using finite element analysis (FEA). We introduce a comprehensive evaluation scheme to investigate the ability of LLMs to solve these problems end-to-end by reasoning over natural language problem descriptions and operating COMSOL Multiphysics$^\circledR$, an FEA software, to compute the answers. We additionally design a language model agent equipped with the ability to interact with the software through its Application Programming Interface (API), examine its outputs and use tools to improve its solutions over multiple iterations. Our best performing strategy generates executable API calls 88% of the time. LLMs that can successfully interact with and operate FEA software to solve problems such as those in our benchmark would push the frontiers of automation in engineering. Acquiring this capability would augment LLMs' reasoning skills with the precision of numerical solvers and advance the development of autonomous systems that can tackle complex problems in the real world. The code is available at https://github.com/google/feabench

Figures (7)

• Figure 1: Left: Illustrative abbreviated example of Model Specifications for one of the heat transfer problems. Right: Distribution of FEABench Gold problems by physics domain.
• Figure 2: An overview of the agent and environment design.
• Figure 4: Block-wise executability across 300 samples of code and Gemini-1.5-Pro. The physics block has the lowest executability. Error bars denote standard deviations.
• Figure 5: Screenshot of the graphical user interface for the correctly solved problem in Figure \ref{['fig:schema']}.
• Figure 6: Block-wise executability across the 300 initial samples of code (purple) with PhyDoc In-Context and in the best solution (green) across all problems. Error bars denote standard deviations.