DSCodeBench: A Realistic Benchmark for Data Science Code Generation

TL;DR

DSCodeBench introduces a realistic, large-scale benchmark for data science code generation by constructing 1000 end-to-end problems from GitHub across ten libraries, with a rigorous pipeline for ground-truth code, automated test-case generation, and careful alignment. The framework yields longer, library-aware tasks with robust test suites, enabling more reliable evaluation than prior benchmarks. Empirical results show a clear scaling trend across models, with GPT-4o achieving the highest pass@1 (0.392) and open-source models lagging behind, underscoring substantial room for improvement in real-world data science coding. Overall, DSCodeBench offers a rigorous, practical evaluation platform that better mirrors real-world workflows and can guide future model development and benchmarking efforts.

Abstract

We introduce DSCodeBench, a new benchmark designed to evaluate large language models (LLMs) on complicated and realistic data science code generation tasks. DSCodeBench consists of 1,000 carefully constructed problems sourced from realistic problems from GitHub across ten widely used Python data science libraries. DSCodeBench offers a more challenging and representative testbed, more complex code solutions, more comprehensive data science libraries, clearer and better structured problem descriptions, and stronger test suites. To construct the DSCodeBench, we develop a robust pipeline that combines task scope selection, code construction, test case generation, and problem description synthesis. The process is paired with rigorous manual editing to ensure alignment and enhance the reliability of the evaluation. Experimental result shows that DSCodeBench exhibits robust scaling behavior, where larger models systematically outperform smaller ones, validating its ability to distinguish model capabilities. The best LLM we test, GPT-4o, has a pass@1 of 0.392, indicating that LLMs still have a large room to improve for realistic data science code generation tasks. We believe DSCodeBench will serve as a rigorous and trustworthy foundation for advancing LLM-based data science programming.

Figures (5)

• Figure 1: An example of DSCodeBench construction. The pipeline begins by determining the task scope, followed by collecting seed code and constructing ground truth code. This ground truth code is then used to generate a corresponding test case script that produces tailored input–output examples. Using both the ground truth and the generated test cases, the problem description is generated. Finally, all components are manually reviewed and aligned.
• Figure 2: Distribution of tasks in DSCodeBench by library.
• Figure 3: Similarity between LLM-generated solution and ground truth code by different libraries.
• Figure 4: Similarity between LLM-generated solution and ground truth code by different models.
• Figure 5: Mean test case coverage on each library.