FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights

TL;DR

FIRE-Bench addresses the challenge of evaluating autonomous, LLM-driven agents across the full scientific research cycle by rediscovering established empirical findings from recent ML literature. The benchmark constructs 30 tasks from high-impact papers using a constrained rediscovery framework built on a hierarchical research-problem tree and automated extraction, with claim-level $F_1$ scoring against ground-truth findings. Across multiple frontier LLM backbones, agents exhibit limited average performance (< $F_1$ of 50) and high variability, with major failures traced to planning and conclusion formation rather than coding; cost analyses reveal trade-offs between model capability and resource usage. The work provides a diagnostic, end-to-end evaluation framework that clearly identifies where current autonomous research agents struggle and guides future improvements in planning, experimental design, and evidence-based reasoning. FIRE-Bench thus offers a principled path toward reliable, interpretable AI-driven scientific discovery and robust benchmarks to track progress.

Abstract

Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge. Existing benchmarks face a trade-off: they either heavily rely on LLM-as-judge evaluations of automatically generated research outputs or optimize convenient yet isolated performance metrics that provide coarse proxies for scientific insight. To address this gap, we introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings from recent, high-impact machine learning research. Agents are given only a high-level research question extracted from a published, verified study and must autonomously explore ideas, design experiments, implement code, execute their plans, and derive conclusions supported by empirical evidence. We evaluate a range of state-of-the-art agents with frontier LLMs backbones like gpt-5 on FIRE-Bench. Our results show that full-cycle scientific research remains challenging for current agent systems: even the strongest agents achieve limited rediscovery success (<50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning. FIRE-Bench provides a rigorous and diagnostic framework for measuring progress toward reliable agent-driven scientific discovery.

Figures (4)

• Figure 1: Comparing FIRE-Bench with other types of benchmarks for research automation, including full paper generation lu2024aiyamada2025ai, isolated stage automation (e.g., ideation, execution, etc.), and method replication & engineering starace2025paperbenchchan2024mle.
• Figure 2: FIRE-Bench presents an AI research agent with a high-level research question from a published study and evaluates its ability to autonomously rediscover the study’s central empirical finding. This formulation enables fine-grained comparison of the agent’s end-to-end research process with the original human workflow.
• Figure 3: Error Distribution of four evaluated agents. Different agents exhibit similar error distributions, with failures in Research Planning and Conclusion Formation accounting for the majority of errors.
• Figure 4: Agent performance stratified by difficulty level. The clear monotonic relationship supports the validity of the proposed difficulty measure.