Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HeurekaBench: A Benchmarking Framework for AI Co-scientist

Siba Smarak Panigrahi, Jovana Videnović, Maria Brbić

TL;DR

This paper tackles the challenge of evaluating AI co-scientists by introducing HeurekaBench, a benchmarking framework grounded in real scientific workflows and validated code. It grounds benchmark construction in published studies to generate open-ended research questions (OEQs) and multiple-choice questions (MCQs), requiring end-to-end reasoning over experimental data. The sc-HeurekaBench instantiation for single-cell biology yields 50 OEQs and 50 MCQs across 41 validated insights, enabling quantitative comparisons of state-of-the-art single-cell agents. The framework provides an evaluation scheme using LLM-based judges and human alignment, and demonstrates that adding a critic module can substantially improve performance on open-ended tasks and reduce gaps between open- and closed-source models. The work points to broad applicability to other scientific domains and highlights future directions for validating intermediate steps and refining open-source agent design.

Abstract

LLM-based reasoning models have enabled the development of agentic systems that act as co-scientists, assisting in multi-step scientific analysis. However, evaluating these systems is challenging, as it requires realistic, end-to-end research scenarios that integrate data analysis, interpretation, and the generation of new insights from the experimental data. To address this limitation, we introduce HeurekaBench, a framework to create benchmarks with exploratory, open-ended research questions for experimental datasets. Each such question is grounded in a scientific study and its corresponding code repository, and is created using a semi-automated pipeline that leverages multiple LLMs to extract insights and generate candidate workflows, which are then verified against reported findings. We instantiate the framework in single-cell biology to obtain sc-HeurekaBench benchmark and use it to compare state-of-the-art single-cell agents. We further showcase the benefits of our benchmark for quantitatively analyzing current design choices in agentic systems. We find that the addition of a critic module can improve ill-formed responses for open-source LLM-based agents by up to 22% and close the gap with their closed-source counterparts. Overall, HeurekaBench sets a path toward rigorous, end-to-end evaluation of scientific agents, grounding benchmark construction in real scientific workflows.

HeurekaBench: A Benchmarking Framework for AI Co-scientist

TL;DR

This paper tackles the challenge of evaluating AI co-scientists by introducing HeurekaBench, a benchmarking framework grounded in real scientific workflows and validated code. It grounds benchmark construction in published studies to generate open-ended research questions (OEQs) and multiple-choice questions (MCQs), requiring end-to-end reasoning over experimental data. The sc-HeurekaBench instantiation for single-cell biology yields 50 OEQs and 50 MCQs across 41 validated insights, enabling quantitative comparisons of state-of-the-art single-cell agents. The framework provides an evaluation scheme using LLM-based judges and human alignment, and demonstrates that adding a critic module can substantially improve performance on open-ended tasks and reduce gaps between open- and closed-source models. The work points to broad applicability to other scientific domains and highlights future directions for validating intermediate steps and refining open-source agent design.

Abstract

LLM-based reasoning models have enabled the development of agentic systems that act as co-scientists, assisting in multi-step scientific analysis. However, evaluating these systems is challenging, as it requires realistic, end-to-end research scenarios that integrate data analysis, interpretation, and the generation of new insights from the experimental data. To address this limitation, we introduce HeurekaBench, a framework to create benchmarks with exploratory, open-ended research questions for experimental datasets. Each such question is grounded in a scientific study and its corresponding code repository, and is created using a semi-automated pipeline that leverages multiple LLMs to extract insights and generate candidate workflows, which are then verified against reported findings. We instantiate the framework in single-cell biology to obtain sc-HeurekaBench benchmark and use it to compare state-of-the-art single-cell agents. We further showcase the benefits of our benchmark for quantitatively analyzing current design choices in agentic systems. We find that the addition of a critic module can improve ill-formed responses for open-source LLM-based agents by up to 22% and close the gap with their closed-source counterparts. Overall, HeurekaBench sets a path toward rigorous, end-to-end evaluation of scientific agents, grounding benchmark construction in real scientific workflows.
Paper Structure (43 sections, 5 figures, 7 tables)

This paper contains 43 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Limitations of existing benchmarks.
  3. Related Work
  4. HeurekaBench Framework
  5. Co-Scientist Task Overview
  6. HeurekaBench Creation
  7. Evaluation
  8. Instantiating HeurekaBench in single-cell biology
  9. sc-HeurekaBench-ToolUsage Benchmark
  10. Experiments
  11. Evaluation of Insight Construction
  12. Evaluation of Single-Cell Biology agents
  13. Impact of Design choices on Co-scientist
  14. Planner ablations
  15. Inter-rater alignment study between LLM-based judges.
  16. ...and 28 more sections

Figures (5)

  • Figure 1: Illustration of the HeurekaBench framework. The HeurekaBench consists of three stages: (a) insight generation, where candidate insights are extracted from scientific articles and semi-automatically validated; (b) question generation, where validated insights are reformulated as question-answer pairs; and (c) question solving, where the agent autonomously designs and executes a multi-step analysis, producing a data-driven answer that is evaluated against published findings.
  • Figure 2: (a) Evaluation of the InsightExtractor module. Number of insights related to expert findings in FlyBase and BixBench. (b) Evaluation of the CodeDescriber and CodeMatcher modules. Number of insights per number of incorrectly retrieved files. $0$ indicates all files retrieved correctly. Red indicates failure cases.
  • Figure 3: Rankings of the three best-performing planner LLMs within the Biomni agent as per three closed-source LLM-based judges. The plot shows that the three judges agree on the performance of planner models and select Claude-4-Sonnet as the best planner model.
  • Figure 4: Question distribution per task category. There are six different categories, with the distribution across both versions of the benchmark following a similar distribution. The questions related to cell-cell communication analysis are obtained only from Li2024UterineNK, which is not part of the sc-HeurekaBench-Lite.
  • Figure 5: Score distribution per task category. Both versions of the benchmark exhibit a similar trend in score distributions across three evaluated LLMs, with the closed-source model outperforming the open-source model in all categories. Cell–cell communication category is sourced exclusively from Li2024UterineNK, which is not included in sc-HeurekaBench-Lite.