A System for Quantifying Data Science Workflows with Fine-Grained Procedural Logging and a Pilot Study

TL;DR

DataInquirer provides a fine-grained, end-to-end logging framework for computational notebooks to quantify how data scientists turn raw data into insights. By capturing incremental cell executions and enabling AST- and replay-based analyses, the method reveals substantial semantic, syntactic, and methodological variation across practitioners analyzing the same data, and shows AI-assisted coding tools can steer workflows. The pilot studies across Chicago traffic and flight-delay tasks demonstrate patterns in data cleaning, exploration, and modeling that differ by expertise and tool use, with LLMs reducing coding burden and sometimes funneling analysis paths. These findings have practical implications for designing robust educational tools, auditing data-science work, and building systems that accommodate diverse analytical approaches while preserving reproducibility.

Abstract

It is important for researchers to understand precisely how data scientists turn raw data into insights, including typical programming patterns, workflow, and methodology. This paper contributes a novel system, called DataInquirer, that tracks incremental code executions in Jupyter notebooks (a type of computational notebook). The system allows us to quantitatively measure timing, workflow, and operation frequency in data science tasks without resorting to human annotation or interview. In a series of pilot studies, we collect 97 traces, logging data scientist activities across four studies. While this paper presents a general system and data analysis approach, we focus on a foundational sub-question in our pilot studies: How consistent are different data scientists in analyzing the same data? We taxonomize variation between data scientists on the same dataset according to three categories: semantic, syntactic, and methodological. Our results suggest that there are statistically significant differences in the conclusions reached by different data scientists on the same task and present quantitative evidence for this phenomenon. Furthermore, our results suggest that AI-powered code tools subtly influence these results, allowing student participants to generate workflows that more resemble expert data practitioners.

Figures (21)

• Figure 1: Outline of the components of DataInquirer
• Figure 2: Example of the stored log structure
• Figure 3: A timeline of cell executions in Flight v1
• Figure 4: Procedure for analysis of variation over logs
• Figure 5: Experts generally write less code and execute it less