EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants

TL;DR

This work introduces EAIRA, a holistic methodology for evaluating AI models as scientific research assistants by combining four complementary techniques: MCQ benchmarks, Open Response benchmarks, lab-style experiments, and field-style experiments. It emphasizes cross-cutting considerations of safety, trust, and reliable uncertainty quantification, and details scalable software infrastructure (STaR) to run large-scale evaluations on HPC systems. The paper also presents domain-specific and multi-domain benchmarks (Astronomy, Climate, AI4S) and open-response benchmarks (SciCode, ALDbench), plus end-to-end evaluation approaches that reveal strengths and weaknesses across real-world scientific tasks. Collectively, these contributions aim to provide a rigorous, adaptable framework for measuring LLMs’ scientific knowledge, reasoning, and safety, and to guide future improvements and benchmarking across domains. The significance lies in enabling scalable, trust-conscious evaluation of AI assistants in critical scientific workflows, with concrete benchmarks, infrastructure, and forward-looking plans for expansion and refinement.

Abstract

Recent advancements have positioned AI, and particularly Large Language Models (LLMs), as transformative tools for scientific research, capable of addressing complex tasks that require reasoning, problem-solving, and decision-making. Their exceptional capabilities suggest their potential as scientific research assistants but also highlight the need for holistic, rigorous, and domain-specific evaluation to assess effectiveness in real-world scientific applications. This paper describes a multifaceted methodology for Evaluating AI models as scientific Research Assistants (EAIRA) developed at Argonne National Laboratory. This methodology incorporates four primary classes of evaluations. 1) Multiple Choice Questions to assess factual recall; 2) Open Response to evaluate advanced reasoning and problem-solving skills; 3) Lab-Style Experiments involving detailed analysis of capabilities as research assistants in controlled environments; and 4) Field-Style Experiments to capture researcher-LLM interactions at scale in a wide range of scientific domains and applications. These complementary methods enable a comprehensive analysis of LLM strengths and weaknesses with respect to their scientific knowledge, reasoning abilities, and adaptability. Recognizing the rapid pace of LLM advancements, we designed the methodology to evolve and adapt so as to ensure its continued relevance and applicability. This paper describes the methodology state at the end of February 2025. Although developed within a subset of scientific domains, the methodology is designed to be generalizable to a wide range of scientific domains.

Figures (18)

• Figure 1: The AGIL approach to generate scalable MCQ benchmarks. The current version of the AI4S benchmark contains only manually accepted MCQs. The AGIL approach enables the integration of automatically accepted MCQs after the validation of their difficulty and quality.
• Figure 2: The performance of various LLMs on SciCode problems. This histogram displays the accuracy (vertical axis, 0% to 100%) of various state-of-the-art LLMs (listed on the horizontal axis) in solving both main problems (red) and their associated subproblems (blue) within SciCode. To solve a main problem, LLMs must implement one Python function per subproblem and integrate them into a comprehensive solution. SciCode provides gold-standard solutions and multiple test cases for reliable automatic evaluation. These consistently poor results highlight the need for LLMs that incorporate scientific knowledge and advanced reasoning to better assist researchers.
• Figure 3: Distribution of the mean scores of GPT-4o responses to all questions in the ALDbench benchmark.
• Figure 4: Example of multi-turn interaction between a researcher and several LLMs used as research assistants in an attempt to repeat the research developed for the HPDC24 paper. Because of space limitations, the figure does not show the models' responses and research analysis. The full interaction for the HPDC24 experiment can be found here: https://tinyurl.com/yv4awky3
• Figure 5: A partial scoring of several models used as AI assistants on August 20, 2024, to solve the zero-overhead checkpointing problem. The results highlight the strengths and weaknesses of different models for the different research steps. We note that the RAG model (Perplexity Pro) has a decisive advantage in several steps for this particular problem. Other models struggle in most steps.