VDSAgents: A PCS-Guided Multi-Agent System for Veridical Data Science Automation

TL;DR

VDSAgents addresses trustworthiness gaps in LLM-driven data science by embedding Veridical Data Science principles into a PCS-guided multi-agent DSLC framework. A central PCS-Agent audits and guides five stage-specific agents across problem definition, data cleaning/EDA, modeling, and evaluation, leveraging perturbation analysis and unit testing for reproducibility. Empirical results on nine datasets show superior execution stability and predictive effectiveness over AutoKaggle and DataInterpreter when using GPT-4o and DeepSeek-V3 backends, with notable gains on noisy or high-dimensional tasks. The work offers a principled, extensible pathway toward trustworthy AI-assisted data analysis with broad implications for automated scientific workflows.

Abstract

Large language models (LLMs) become increasingly integrated into data science workflows for automated system design. However, these LLM-driven data science systems rely solely on the internal reasoning of LLMs, lacking guidance from scientific and theoretical principles. This limits their trustworthiness and robustness, especially when dealing with noisy and complex real-world datasets. This paper provides VDSAgents, a multi-agent system grounded in the Predictability-Computability-Stability (PCS) principles proposed in the Veridical Data Science (VDS) framework. Guided by PCS principles, the system implements a modular workflow for data cleaning, feature engineering, modeling, and evaluation. Each phase is handled by an elegant agent, incorporating perturbation analysis, unit testing, and model validation to ensure both functionality and scientific auditability. We evaluate VDSAgents on nine datasets with diverse characteristics, comparing it with state-of-the-art end-to-end data science systems, such as AutoKaggle and DataInterpreter, using DeepSeek-V3 and GPT-4o as backends. VDSAgents consistently outperforms the results of AutoKaggle and DataInterpreter, which validates the feasibility of embedding PCS principles into LLM-driven data science automation.

Figures (7)

• Figure 1: Illustration of the DSLC. It includes six stages: (1) identifying and formulating domain problems and collecting data; (2) data cleaning, pre-processing, and early exploration; (3) optional structural analysis and data mining; (4) optional modeling and statistical inference; (5) evaluation and validation of results; and (6) interpretation, communication, and domain knowledge update.
• Figure 2: Workflow architecture of the VDSAgents. Note that the PCS-Agent interacts with all stage-specific agents to evaluate predictability, computability, and stability.
• Figure 3: Comparison of Comprehensive Score (CS) across four system variants on nine tasks
• Figure 4: Debugging workflow between the code executor and LLM-based repair module.
• Figure 5: ANPS comparison under GPT-4o backend with standard deviation error bars.