BLADE: Benchmarking Language Model Agents for Data-Driven Science

TL;DR

BLADE addresses the challenge of evaluating language model agents on open-ended, data-driven scientific analyses by constructing a ground-truth–driven benchmark that pairs real-world research questions with expert analyses and automatic evaluation. It introduces representations for analytical decisions (conceptual variables, data transformations, and statistical models) and matching procedures (value and fuzzy-graph isomorphism) to automatically assess agent outputs. Experiments reveal that while LMs can generate basic analyses, they exhibit limited diversity and modeling sophistication, whereas agents that interact with data via a ReAct framework show improved coverage but still fall short of optimality. The work provides a rigorous, open-source foundation for measuring and improving data-driven scientific reasoning in LM agents.

Abstract

Data-driven scientific discovery requires the iterative integration of scientific domain knowledge, statistical expertise, and an understanding of data semantics to make nuanced analytical decisions, e.g., about which variables, transformations, and statistical models to consider. LM-based agents equipped with planning, memory, and code execution capabilities have the potential to support data-driven science. However, evaluating agents on such open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions. To address these challenges, we present BLADE, a benchmark to automatically evaluate agents' multifaceted approaches to open-ended research questions. BLADE consists of 12 datasets and research questions drawn from existing scientific literature, with ground truth collected from independent analyses by expert data scientists and researchers. To automatically evaluate agent responses, we developed corresponding computational methods to match different representations of analyses to this ground truth. Though language models possess considerable world knowledge, our evaluation shows that they are often limited to basic analyses. However, agents capable of interacting with the underlying data demonstrate improved, but still non-optimal, diversity in their analytical decision making. Our work enables the evaluation of agents for data-driven science and provides researchers deeper insights into agents' analysis approaches.

Figures (22)

• Figure 1: Overview of BLADE. We gathered research questions and datasets from existing research papers, crowd-sourced analysis studies and statistic textbooks as well as analyses from expert annotators (boxes 1-2-3, and Sec. \ref{['sec:benchmark_consturction']}). Given a research question and dataset, LM agents generate a full analysis containing the relevant conceptual variables, a data transform function, and a statistical modeling function (boxes 1-4-5, and Sec. \ref{['sec:task_gen']}). BLADE automatically evaluates this against the ground truth (box 6 and Sec. \ref{['sec:enable_eval']}).
• Figure 2: To allow flexible and fine-grained matching, we represent transforms in code (left) as a column data flow graph $G$ (right). The nodes in blue are column indicator nodes $P$, and the nodes in orange are transform nodes $T$. Details of the data flow graph formalization are in Appendix \ref{['appendix:matching_def']}
• Figure 3: Accuracy scores and 95% confidence intervals for different models on BLADE's 188 MCQs (168 for transformations and 20 for conceptual variables).
• Figure 4: Average precision (top row) and coverage@10 (bottom row) percentages averaged across datasets in BLADE. All runs were included in the results. Run errors default to a hit rate of 0 and are counted in the coverage calculation (i.e., treated as a run that generated nothing). Error bars represent bootstrapped 95% confidence intervals.
• Figure 5: Characterization of run results for analysis generation for each LM and ReAct agent variants. "No execution errors" indicates executable transform code, "Empty transform" means no transformations were provided, "Execution errors" means the code resulted in errors, and "No generation" indicates the result could not be parsed.