Benchmarking AI Performance on End-to-End Data Science Projects

TL;DR

The paper addresses the gap that AI benchmarks often assess isolated tasks, not integrated data science workflows. It builds a 40-project end-to-end benchmark with a 17-category rubric ( totaling 45 points) and an automated grading pipeline powered by LLMs to compare AI-generated projects across seven models. Findings show that top models approach undergraduate performance on routine tasks but exhibit substantial variation on judgment-heavy components such as measurement, abstract writing, and citations, while template-driven sections are more reliably handled. The work highlights the need for verification, domain-specific benchmarking, and careful consideration before deploying AI for autonomous data science work.

Abstract

Data science is an integrated workflow of technical, analytical, communication, and ethical skills, but current AI benchmarks focus mostly on constituent parts. We test whether AI models can generate end-to-end data science projects. To do this we create a benchmark of 40 end-to-end data science projects with associated rubric evaluations. We use these to build an automated grading pipeline that systematically evaluates the data science projects produced by generative AI models. We find the extent to which generative AI models can complete end-to-end data science projects varies considerably by model. Most recent models did well on structured tasks, but there were considerable differences on tasks that needed judgment. These findings suggest that while AI models could approximate entry-level data scientists on routine tasks, they require verification.

Figures (10)

• Figure 1: Overall performance, both score and percentage, of seven AI models on an end-to-end data analysis project. The maximum score is 45, and the horizontal dashed line at 36 (80 per cent) is the threshold between a B+ and A-, which is the typical performance of good upper-year undergraduates.
• Figure 2: Model performance by rubric evaluation type shown as the percentage of maximum possible points for that type. Evaluation types and maximum number of points are: Reproducibility (9 points), Presentation (10 points), Analysis (10 points), Graphs (8 points), Referencing (4 points), and Writing (4 points).
• Figure 3: Model performance for all 17 rubric categories shown as percentage of maximum possible points for that category. Categories are ordered by average performance across all models (highest at top). Each point is one model's average performance.
• Figure 4: Distribution of individual project scores for each model. Each point represents one of five projects. Models are ordered by mean score.
• Figure 5: Claude Opus 4.6, Project 5 (87%)