Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ALL-FEM: Agentic Large Language models Fine-tuned for Finite Element Methods

Rushikesh Deotale, Adithya Srinivasan, Yuan Tian, Tianyi Zhang, Pavlos Vlachos, Hector Gomez

Abstract

Finite element (FE) analysis guides the design and verification of nearly all manufactured objects. It is at the core of computational engineering, enabling simulation of complex physical systems, from fluids and solids to multiphysics systems. However, implementing FE codes and analyzing simulation results demands expertise across numerical analysis, continuum mechanics, and programming. Conventional Large Language Models (LLMs) can generate FE code, but they hallucinate, lack awareness of variational structures, and cannot close the loop from problem statement to a verified solution. Here, we propose ALL-FEM, an autonomous simulation system that integrates agentic AI with domain-specific, fine-tuned LLMs for FEniCS code generation across solid, fluid, and multiphysics applications. We construct a corpus of 1000+ verified FEniCS scripts by combining 500+ curated expert codes with a retrieval-augmented, multi-LLM pipeline that generates and filters codes for diverse PDEs, geometries, and boundary conditions. We used the corpus to fine-tune LLMs with 3B to 120B parameters. Our agentic framework orchestrates specialized agents, powered by fine-tuned LLMs, to formulate problems as PDEs, generate and debug code and visualize the results. We evaluated the system on 39 benchmarks that include problems of linear/nonlinear elasticity, plasticity, Newtonian/non-Newtonian flow, thermofluids, fluid-structure interaction, phase separation, and transport on moving domains. Embedded in a multi-agent workflow with runtime feedback, the best fine-tuned model (GPT OSS 120B) achieves code-level success of 71.79%, outperforming a non-agentic deployment of GPT 5 Thinking. By showing that relatively small, fine-tuned LLMs, orchestrated through agentic frameworks, can automate FE workflows, ALL-FEM offers a blueprint for autonomous simulation systems in computational science and engineering.

ALL-FEM: Agentic Large Language models Fine-tuned for Finite Element Methods

Abstract

Finite element (FE) analysis guides the design and verification of nearly all manufactured objects. It is at the core of computational engineering, enabling simulation of complex physical systems, from fluids and solids to multiphysics systems. However, implementing FE codes and analyzing simulation results demands expertise across numerical analysis, continuum mechanics, and programming. Conventional Large Language Models (LLMs) can generate FE code, but they hallucinate, lack awareness of variational structures, and cannot close the loop from problem statement to a verified solution. Here, we propose ALL-FEM, an autonomous simulation system that integrates agentic AI with domain-specific, fine-tuned LLMs for FEniCS code generation across solid, fluid, and multiphysics applications. We construct a corpus of 1000+ verified FEniCS scripts by combining 500+ curated expert codes with a retrieval-augmented, multi-LLM pipeline that generates and filters codes for diverse PDEs, geometries, and boundary conditions. We used the corpus to fine-tune LLMs with 3B to 120B parameters. Our agentic framework orchestrates specialized agents, powered by fine-tuned LLMs, to formulate problems as PDEs, generate and debug code and visualize the results. We evaluated the system on 39 benchmarks that include problems of linear/nonlinear elasticity, plasticity, Newtonian/non-Newtonian flow, thermofluids, fluid-structure interaction, phase separation, and transport on moving domains. Embedded in a multi-agent workflow with runtime feedback, the best fine-tuned model (GPT OSS 120B) achieves code-level success of 71.79%, outperforming a non-agentic deployment of GPT 5 Thinking. By showing that relatively small, fine-tuned LLMs, orchestrated through agentic frameworks, can automate FE workflows, ALL-FEM offers a blueprint for autonomous simulation systems in computational science and engineering.
Paper Structure (26 sections, 2 equations, 6 figures)

This paper contains 26 sections, 2 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. LLM Selection
  4. Fine-tuning
  5. Dataset Creation
  6. Seed Data Collection
  7. Synthetic Data Generation Pipeline
  8. Low-Rank Adaptation (LoRA) and Quantized LoRA (QLoRA)
  9. Hyperparameters and Hardware
  10. LoRA Settings
  11. Learning Optimizer Settings
  12. Hardware
  13. Validation
  14. Agentic AI
  15. Two-agent Framework
  16. ...and 11 more sections

Figures (6)

  • Figure 1: Flowchart describing our data augmentation technique. We use a multi-model pipeline to create synthetic FEniCS codes from a seed dataset.
  • Figure 2: Evaluation mean token accuracy (top row) and training loss (bottom row) in a 5-fold cross-validation of our fine-tuning process. The left, center and right columns show results for Llama 3.2 3B, Qwen 3 32B, and Llama 3.3 70B, respectively.
  • Figure 3: Two-agent framework: the FEniCS Coder (Assistant Agent) generates code from the prompt. The Executor (User Proxy Agent) runs the code, and any errors are fed back to the coder until execution succeeds.
  • Figure 4: Schematic of the multi-agent framework. The Coordinator manages agent interactions. The agent types are identified by the type of box. For those agents powered by an LLM, we write the name of the model in their corresponding box. The yellow agents are directly eligible by the Coordinator, while the green agents are not.
  • Figure 5: Performance comparison of agentic LLM frameworks and GPT-5 across difficulty levels. Bars represent the percentage of questions solved correctly, while numerals over bar indicate the number of questions solved correctly.
  • ...and 1 more figures