DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems

TL;DR

DSAEval introduces a comprehensive, real-world benchmark for autonomous data science agents spanning 285 datasets and 641 tasks across structured, unstructured, and multimodal domains. It combines a Multimodal Environment Perception framework, Multi-Query Interactions, and a Multi-Dimensional Evaluation protocol driven by LLM-based judges to score reasoning, code, and results with weights $\alpha=0.3$, $\beta=0.3$, and $1-\alpha-\beta=0.4$. The evaluation of 11 advanced models shows that larger, multimodal-capable models achieve superior overall performance, with Claude-Sonnet-4.5 leading, GPT-5.2 being the most efficient, and MiMo-V2-Flash the most cost-effective. Multimodal perception consistently boosts vision-related task performance by up to $11.30\%$, while current agents still struggle with unstructured data tasks like CV/NLP and complex modeling workflows. The paper also outlines future directions to broaden dataset coverage, shift evaluation toward holistic agent systems, and expand the benchmark to guide next-generation data science agents.

Abstract

Recent LLM-based data agents aim to automate data science tasks ranging from data analysis to deep learning. However, the open-ended nature of real-world data science problems, which often span multiple taxonomies and lack standard answers, poses a significant challenge for evaluation. To address this, we introduce DSAEval, a benchmark comprising 641 real-world data science problems grounded in 285 diverse datasets, covering both structured and unstructured data (e.g., vision and text). DSAEval incorporates three distinctive features: (1) Multimodal Environment Perception, which enables agents to interpret observations from multiple modalities including text and vision; (2) Multi-Query Interactions, which mirror the iterative and cumulative nature of real-world data science projects; and (3) Multi-Dimensional Evaluation, which provides a holistic assessment across reasoning, code, and results. We systematically evaluate 11 advanced agentic LLMs using DSAEval. Our results show that Claude-Sonnet-4.5 achieves the strongest overall performance, GPT-5.2 is the most efficient, and MiMo-V2-Flash is the most cost-effective. We further demonstrate that multimodal perception consistently improves performance on vision-related tasks, with gains ranging from 2.04% to 11.30%. Overall, while current data science agents perform well on structured data and routine data anlysis workflows, substantial challenges remain in unstructured domains. Finally, we offer critical insights and outline future research directions to advance the development of data science agents.

Figures (7)

• Figure 1: Overall performance of all models on DSAEval.
• Figure 2: Overview of DSAEval. Left: In the Data Collection Pipeline, raw cases are cleaned and synthesized into Question, Reasoning, and Answer (QRA) pairs using advanced LLMs. Middle: The Data Agent Pipeline orchestrates the agent to solve tasks within a Sandbox Environment. The agent receives multimodal observations and produces a final report and a Jupyter notebook. Right: The Multi-Dimensional Evaluation module employs a Judge model to score the reasoning, code, and results against the soft ground truth, yielding a composite final score.
• Figure 3: Distribution of the DSAEval benchmark. The suite covers diverse data modalities (left), problem domains (center), and task types (right), ensuring comprehensive evaluation coverage.
• Figure 4: Fine-grained Performance Analysis. Left: Performance by Domain shows robust capabilities in Data Analysis but significant weaknesses in Computer Vision and NLP. Right: Performance by Task Type highlights proficiency in Data Ingestion but bottlenecks in Statistical Inference and Model Evaluation.
• Figure 5: Efficiency and Cost-Effectiveness Analysis. Left: Total Score vs. Average Tokens. Right: Total Score vs. Average Price Per Task. The closer to the top left corner, the better.