DSGym: A Holistic Framework for Evaluating and Training Data Science Agents

TL;DR

DSGym offers a unified, execution-grounded framework to evaluate data science agents across diverse tasks with real data in isolated, reproducible environments. It standardizes task representation, agent interfaces, and runtime infrastructure, and expands coverage with DSBio and DSPredict to probe domain grounding and end-to-end modeling. The framework includes a two-stage task refinement process to remove shortcut solvability and ensure data-dependent reasoning, while also enabling training via execution-verified synthetic data (DSGym-SFT). Empirical results reveal persistent domain-specific grounding gaps and a tendency toward simplicity bias, yet demonstrate that DSGym can support data-efficient training that rivals frontier models on standard benchmarks. Overall, DSGym functions as a live, extensible testbed for measuring and advancing data science agents in realistic scientific contexts.

Abstract

Data science agents promise to accelerate discovery and insight-generation by turning data into executable analyses and findings. Yet existing data science benchmarks fall short due to fragmented evaluation interfaces that make cross-benchmark comparison difficult, narrow task coverage and a lack of rigorous data grounding. In particular, we show that a substantial portion of tasks in current benchmarks can be solved without using the actual data. To address these limitations, we introduce DSGym, a standardized framework for evaluating and training data science agents in self-contained execution environments. Unlike static benchmarks, DSGym provides a modular architecture that makes it easy to add tasks, agent scaffolds, and tools, positioning it as a live, extensible testbed. We curate DSGym-Tasks, a holistic task suite that standardizes and refines existing benchmarks via quality and shortcut solvability filtering. We further expand coverage with (1) DSBio: expert-derived bioinformatics tasks grounded in literature and (2) DSPredict: challenging prediction tasks spanning domains such as computer vision, molecular prediction, and single-cell perturbation. Beyond evaluation, DSGym enables agent training via execution-verified data synthesis pipeline. As a case study, we build a 2,000-example training set and trained a 4B model in DSGym that outperforms GPT-4o on standardized analysis benchmarks. Overall, DSGym enables rigorous end-to-end measurement of whether agents can plan, implement, and validate data analyses in realistic scientific context.

Figures (10)

• Figure 1: (a) In the typical scientific discovery process, DSGym specifically focuses on the Data-Driven Investigation phase, where agents must bridge scientific hypotheses and empirical evidence through complex analysis. (b) We provide a unified environment spanning 10+ scientific domains and diverse file types. The framework enables a closed-loop ecosystem for both evaluation and training.
• Figure 2: The Architecture of DSGym. (a) Standardized Tasks: We aggregate heterogeneous data sources into a unified task object. (b) Agent Interface:DSGym provides a default CodeAct-like agent to interact with the environment. (c) Execution Environment: A central Manager container orchestrates the execution. Based on the task type, it dispatches agents to isolated Docker containers (Workers) pre-loaded with domain-specific libraries. Crucially, datasets are mounted as Read-Only Volumes, while agents operate in a separate writable workspace.
• Figure 3: Accuracy with or without data access on three file-grounded benchmarks. We observe that even when real data files are not provided, agents can still answer a substantial fraction of questions correctly, suggesting that existing benchmarks can be partially solved via memorization, pattern matching, or priors rather than genuine data interaction.
• Figure 4: Example questions across data science benchmarks. Existing datasets such as QRData, DAEval, and DABStep mainly target general or applied data-science operations. DSGym complements these with new domain-specific scientific tasks (e.g., bioinformatics) that require specialized workflows and terminology.
• Figure 5: Filtering statistics after two-stage refinement.