Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs

Saeed Mohammadzadeh, Erfan Hamdi, Joel Shor, Emma Lejeune

TL;DR

FEM-Bench addresses the gap in evaluating LLMs' ability to generate physics-based numerical code by introducing a structured, executable benchmark drawn from computational mechanics. The framework pairs modular tasks with reference implementations and unit tests, and uses automated prompting, parsing, and objective VVUQ-style evaluation to measure function correctness and test validity. Results show that while state-of-the-art LLMs can reproduce foundational FEM/MSA components, they struggle with geometry, buckling, and producing discriminative physics-aware tests, underscoring the need for domain knowledge and robust verification in AI-assisted scientific computing. As a living benchmark, FEM-Bench enables systematic tracking of progress and guides future task design toward more challenging, realistic computational mechanics workflows.

Abstract

As LLMs advance their reasoning capabilities about the physical world, the absence of rigorous benchmarks for evaluating their ability to generate scientifically valid physical models has become a critical gap. Computational mechanics, which develops and applies mathematical models and numerical methods to predict the behavior of physical systems under forces, deformation, and constraints, provides an ideal foundation for structured scientific reasoning evaluation. Problems follow clear mathematical structure, enforce strict physical and numerical constraints, and support objective verification. The discipline requires constructing explicit models of physical systems and reasoning about geometry, spatial relationships, and material behavior, connecting directly to emerging AI goals in physical reasoning and world modeling. We introduce FEM-Bench, a computational mechanics benchmark designed to evaluate the ability of LLMs to generate correct finite element method (FEM) and related code. FEM-Bench 2025 contains a suite of introductory but nontrivial tasks aligned with material from a first graduate course on computational mechanics. These tasks capture essential numerical and physical modeling challenges while representing only a small fraction of the complexity present in the discipline. Despite their simplicity, state-of-the-art LLMs do not reliably solve all of them. In a five attempt run, the best performing model at function writing, Gemini 3 Pro, completed 30/33 tasks at least once and 26/33 tasks all five times. The best performing model at unit test writing, GPT-5, had an Average Joint Success Rate of 73.8%. Other popular models showed broad performance variation. FEM-Bench establishes a structured foundation for evaluating AI-generated scientific code, and future iterations will incorporate increasingly sophisticated tasks to track progress as models evolve.

FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs

TL;DR

FEM-Bench addresses the gap in evaluating LLMs' ability to generate physics-based numerical code by introducing a structured, executable benchmark drawn from computational mechanics. The framework pairs modular tasks with reference implementations and unit tests, and uses automated prompting, parsing, and objective VVUQ-style evaluation to measure function correctness and test validity. Results show that while state-of-the-art LLMs can reproduce foundational FEM/MSA components, they struggle with geometry, buckling, and producing discriminative physics-aware tests, underscoring the need for domain knowledge and robust verification in AI-assisted scientific computing. As a living benchmark, FEM-Bench enables systematic tracking of progress and guides future task design toward more challenging, realistic computational mechanics workflows.

Abstract

As LLMs advance their reasoning capabilities about the physical world, the absence of rigorous benchmarks for evaluating their ability to generate scientifically valid physical models has become a critical gap. Computational mechanics, which develops and applies mathematical models and numerical methods to predict the behavior of physical systems under forces, deformation, and constraints, provides an ideal foundation for structured scientific reasoning evaluation. Problems follow clear mathematical structure, enforce strict physical and numerical constraints, and support objective verification. The discipline requires constructing explicit models of physical systems and reasoning about geometry, spatial relationships, and material behavior, connecting directly to emerging AI goals in physical reasoning and world modeling. We introduce FEM-Bench, a computational mechanics benchmark designed to evaluate the ability of LLMs to generate correct finite element method (FEM) and related code. FEM-Bench 2025 contains a suite of introductory but nontrivial tasks aligned with material from a first graduate course on computational mechanics. These tasks capture essential numerical and physical modeling challenges while representing only a small fraction of the complexity present in the discipline. Despite their simplicity, state-of-the-art LLMs do not reliably solve all of them. In a five attempt run, the best performing model at function writing, Gemini 3 Pro, completed 30/33 tasks at least once and 26/33 tasks all five times. The best performing model at unit test writing, GPT-5, had an Average Joint Success Rate of 73.8%. Other popular models showed broad performance variation. FEM-Bench establishes a structured foundation for evaluating AI-generated scientific code, and future iterations will incorporate increasingly sophisticated tasks to track progress as models evolve.
Paper Structure (33 sections, 15 equations, 5 figures, 6 tables)

This paper contains 33 sections, 15 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Computational Mechanics as a Test of Physical World Modeling
  4. Computational Mechanics as a Test of Structured, Multistep Reasoning
  5. Verification and Evaluation in Computational Mechanics
  6. Motivation for FEM-Bench
  7. Methods
  8. Problem Definition
  9. Task Definition and Structure
  10. Example Task
  11. Prompt Generation
  12. Prompting Procedure and Inference Settings
  13. Output Parsing and Validation
  14. Evaluation of Generated Code and Tests
  15. Function Correctness Evaluation
  16. ...and 18 more sections

Figures (5)

  • Figure 1: Schematic overview of the finite element method (FEM). (i) A solid mechanics problem: given loads and boundary conditions on the reference configuration, solve for the deformation mapping to the deformed configuration. (ii) Discretization step: the continuous domain is approximated by a mesh of finite elements and nodes, enabling numerical solution of the relevant governing equations.
  • Figure 2: Overview of the FEM-Bench workflow. Tasks are defined as self-contained Python modules specifying reference implementations, dependencies, and unit tests. The core FEM-Bench software loads these tasks, constructs standardized prompts for both code-generation and test-generation, and interfaces with LLMs through model-specific API clients. LLM outputs (generated code and test suites) are then parsed, validated, and evaluated using reference-based numerical checks and expected-failure unit tests. The framework produces detailed scores for each model and task, enabling reproducible and interpretable comparison of LLM performance.
  • Figure 3: Comparison of model performance on FEM-Bench 2025. The plot shows function correctness versus average joint test success rate for all evaluated models (pulled from Table \ref{['tab:results']} and Table \ref{['tab:joint_test_results']} data), illustrating clear capability differences across model families and identifying the current performance frontier.
  • Figure 4: Schematic comparison of the reference implementations used in FEM-Bench for (i) linear elastic analysis (a solved task) and (ii) elastic critical load analysis (a currently unsolved task). In both cases, the workflow begins by assembling the global load vector $\mathbf{F}$ and global elastic stiffness matrix $\mathbf{K}$ from local element contributions, followed by partitioning degrees of freedom and performing a linear solve. For elastic critical load analysis, the converged displacement field is used to assemble the geometric stiffness matrix $\mathbf{K}_{\mathrm{geom}}$, after which an eigenvalue analysis is performed to compute critical loads. The outputs consist of nodal displacements and reactions for the linear case, and critical load factors and mode shapes for the buckling case. Note that the elastic critical load analysis contains the linear elastic analysis within it.
  • Figure 5: Illustration of the Matrix Structural Analysis workflow. A structural frame is discretized into nodes and elements with associated material and section properties, boundary conditions, and nodal loads. These inputs are used to assemble the global load vector and global stiffness matrix by computing local element stiffnesses and transforming them to global coordinates. The resulting system $\mathbf{F} = \mathbf{K}\boldsymbol{\Delta}$ is then solved for nodal displacements and support reactions.