AInsteinBench: Benchmarking Coding Agents on Scientific Repositories

TL;DR

AInsteinBench addresses the need to evaluate LLM-based agents as genuine scientific computing collaborators within real, production-scale software ecosystems. By constructing a large, executable benchmark from maintainer-authored PRs across six diverse codebases, the work assesses end-to-end tasks—ranging from feature implementation to bug fixes—within containerized environments and test-driven verification. The results reveal meaningful progress in domain-aware reasoning but also persistent gaps in preserving scientific invariants and cross-file consistency, underscoring the uniquely scientific challenges beyond traditional SWE benchmarks. Overall, the benchmark provides a foundation for developing reliable, domain-aware coding agents capable of contributing to modern computational science at scale.

Abstract

We introduce AInsteinBench, a large-scale benchmark for evaluating whether large language model (LLM) agents can operate as scientific computing development agents within real research software ecosystems. Unlike existing scientific reasoning benchmarks which focus on conceptual knowledge, or software engineering benchmarks that emphasize generic feature implementation and issue resolving, AInsteinBench evaluates models in end-to-end scientific development settings grounded in production-grade scientific repositories. The benchmark consists of tasks derived from maintainer-authored pull requests across six widely used scientific codebases, spanning quantum chemistry, quantum computing, molecular dynamics, numerical relativity, fluid dynamics, and cheminformatics. All benchmark tasks are carefully curated through multi-stage filtering and expert review to ensure scientific challenge, adequate test coverage, and well-calibrated difficulty. By leveraging evaluation in executable environments, scientifically meaningful failure modes, and test-driven verification, AInsteinBench measures a model's ability to move beyond surface-level code generation toward the core competencies required for computational scientific research.

Figures (8)

• Figure 1: Overall performance score across frontier models among all difficult tiers
• Figure 2: Scientist vs LLM agent workflow In computational sciences like physics, chemistry and biology, the daily work of a graduate student and postdoc usually consists of such a process: Reading papers and textbooks for understanding of both science and coding; Experimenting with computer simulations or actual machines or laboratories to gain feedback and iterate; Solving problems or implementing new features to a project or the computational code repository it self. Throughout the process the scientist would actively interact with the computing allocations and the school/laboratory resources. On the other hand, in this paper, we setup docker container with terminal access as an environment for the LLM agent to interact with. Allowing the LLM agent to view files, execute bash commands, and commit their changes. Through this processes, the human researches and the LLM agents contribute their ability to solve real-world problems in scientific computing projects across different disciplines including RDKit, Qiskit, EinsteinToolkit, PySCF, OpenMM, and AMReX.
• Figure 3: Overall data curation and benchmark construction pipeline.
• Figure 4: Scientific and engineering difficulty by domain(green: easy, yellow: medium, red: hard).
• Figure 5: Problem resolution rate across difficulty tiers and scientific domains (green: easy, yellow: medium, red: hard). Overall difficulty is defined as the mean of engineering and scientific difficulty