SimBench: A Framework for Evaluating and Diagnosing LLM-Based Digital-Twin Generation for Multi-Physics Simulation

TL;DR

SimBench presents a multi-turn benchmark and dataset to evaluate simulator-focused LLMs in generating simulator-ready digital twins for multi-physics simulations, using a rubric-based J-LLM as an interpretable evaluator and partial execution anchoring. It demonstrates the approach on PyChrono with 102 turn-level tasks across 34 physical systems and 33 S-LLMs, finding a measurable but still headroom-filled improvement trajectory for newer models, and showing that rubric-based judging correlates more with functional correctness (Pass@1) than traditional similarity metrics. The methodology is simulator-agnostic and designed for reproducibility, with open-source artifacts and a clear porting path to other simulation ecosystems. Overall, SimBench provides both diagnostic insights into current S-LLM capabilities and a practical blueprint for advancing simulator-grade DT generation.

Abstract

We introduce SimBench, a benchmark designed to evaluate the proficiency of simulator-oriented LLMs (S-LLMs) in generating digital twins (DTs) that can be used in simulators for virtual testing. Given a collection of S-LLMs, this benchmark ranks them according to their ability to produce high-quality DTs. We demonstrate this by comparing over 33 open- and closed-source S-LLMs. Using multi-turn interactions, SimBench employs an LLM-as-a-judge (J-LLM) that leverages both predefined rules and human-in-the-loop guidance to assign scores for the DTs generated by the S-LLM, thus providing a consistent and expert-inspired evaluation protocol. The J-LLM is specific to a simulator, and herein the proposed benchmarking approach is demonstrated in conjunction with the open-sourceChrono multi-physics simulator. Chrono provided the backdrop used to assess an S-LLM in relation to the latter's ability to create digital twins for multibody dynamics, finite element analysis, vehicle dynamics, robotic dynamics, and sensor simulations. The proposed benchmarking principle is broadly applicable and enables the assessment of an S-LLM's ability to generate digital twins for other simulation packages, e.g., ANSYS, ABAQUS, OpenFOAM, StarCCM+, IsaacSim, and pyBullet.

Figures (4)

• Figure 1: The SimBench pipeline for evaluating S-LLMs. The J-LLM is calibrated using a validation set containing pairs of ground truth and generated DTs. The prompts given to the J-LLM are iteratively optimized to match the score provided by the expert. Then the J-LLM is used to evaluate the S-LLM based on the generated DTs, ground truth DTs, and the API documentation.
• Figure 2: A subset of simulation scenarios in SimBench.
• Figure 3: Temporal evolution of S-LLM performance on SimBench. Y-axis is the average of three modalities of J-LLM scores, they are plotted against model release dates, showing a positive trend ($\rho = 0.624$, $p < 0.001$).
• Figure 4: The correlation matrix of different metrics, including pass@1, compile@1, three different J-LLM instances, and the similarity scores CodeBLEU and ROUGE-LSUM.