DataSciBench: An LLM Agent Benchmark for Data Science

TL;DR

DataSciBench addresses the need for a comprehensive data-science oriented LLM evaluation by introducing a semi-automated ground-truth generation pipeline and a Task-Function-Code (TFC) framework that links task definitions to executable evaluation metrics. The benchmark collects 222 prompts that yield 519 ground-truth test cases across 6 data-science task types, and evaluates 23 LLMs (API-based and open-source) using coarse-grained (CR, SR) and fine-grained (VLM judgments and 25 function metrics) criteria. API-based models generally outperform open-source models, with GPT-4o achieving the highest overall score and certain open-source codes (e.g., Deepseek-Coder-33B-Instruct) performing competitively on some metrics. The work provides detailed analyses, correlations with existing benchmarks, and a public release of code and data to enable broader experimentation and benchmark refinement.

Abstract

This paper presents DataSciBench, a comprehensive benchmark for evaluating Large Language Model (LLM) capabilities in data science. Recent related benchmarks have primarily focused on single tasks, easily obtainable ground truth, and straightforward evaluation metrics, which limits the scope of tasks that can be evaluated. In contrast, DataSciBench is constructed based on a more comprehensive and curated collection of natural and challenging prompts for uncertain ground truth and evaluation metrics. We develop a semi-automated pipeline for generating ground truth (GT) and validating evaluation metrics. This pipeline utilizes and implements an LLM-based self-consistency and human verification strategy to produce accurate GT by leveraging collected prompts, predefined task types, and aggregate functions (metrics). Furthermore, we propose an innovative Task - Function - Code (TFC) framework to assess each code execution outcome based on precisely defined metrics and programmatic rules. Our experimental framework involves testing 6 API-based models, 8 open-source general models, and 9 open-source code generation models using the diverse set of prompts we have gathered. This approach aims to provide a more comprehensive and rigorous evaluation of LLMs in data science, revealing their strengths and weaknesses. Experimental results demonstrate that API-based models outperform open-sourced models on all metrics and Deepseek-Coder-33B-Instruct achieves the highest score among open-sourced models. We release all code and data at https://github.com/THUDM/DataSciBench.

Figures (7)

• Figure 1: This example compares the vanilla response and the DataSciBench response for a given prompt. The vanilla response provides only code, lacking evaluation metrics. In contrast, DataSciBench identifies evaluation tasks, provides evaluation functions, and generates programmatic code to form a TFC list.
• Figure 2: The overall framework of DataSciBench$\space$ encompasses three key components aligned with Section \ref{['sec: DataSciBench']}: 1) Prompt definition and collection, which covers 6 task types, prompt collection, question filtering, and expert review. 2) Response integration and validation, incorporating the TFC framework (25 aggregated functions and programmatic rules) and 519 test cases with ground truth. 3) LLM evaluation involving 23 LLMs.
• Figure 3: Statistics of 6 task types and 25 aggregate functions in Task-Function-Code (TFC) list. DM & PR denotes Data Mining & Pattern Recognition. Interpre. & RG denotes Interpretability & Report Generation.
• Figure 4: Overall score results of all tested LLMs.
• Figure 5: Average Completion Rate results regarding different difficulty levels.