Human to Document, AI to Code: Comparing GenAI for Notebook Competitions

TL;DR

This paper investigates how human-written and GenAI-generated notebooks differ in competitive data science settings. It analyzes three Kaggle competitions, using gold medals as a quality proxy, and compares 465 human notebooks with 9 GenAI notebooks by extracting 25 documentation and code-quality features via SonarQube. The findings show GenAI notebooks exhibit higher static code quality (fewer code smells, technical debt, and violations) while human notebooks deliver more accessible, readable documentation; several GenAI notebooks fail to run, underscoring a gap between static quality and practical usability. The work highlights complementary strengths and argues for hybrid human–GenAI notebook workflows, while outlining a broader research agenda on collaboration, validity, and generalization beyond Kaggle. This establishes a foundation for integrating GenAI into notebook-driven data science while preserving narrative clarity and analytical rigor.

Abstract

Computational notebooks have become the preferred tool of choice for data scientists and practitioners to perform analyses and share results. Notebooks uniquely combine scripts with documentation. With the emergence of generative AI (GenAI) technologies, it is increasingly important, especially in competitive settings, to distinguish the characteristics of human-written versus GenAI. In this study, we present three case studies to explore potential strengths of both humans and GenAI through the coding and documenting activities in notebooks. We first characterize differences between 25 code and documentation features in human-written, medal-winning Kaggle notebooks. We find that gold medalists are primarily distinguished by longer and more detailed documentation. Second, we analyze the distinctions between human-written and GenAI notebooks. Our results show that while GenAI notebooks tend to achieve higher code quality (as measured by metrics like code smells and technical debt), human-written notebooks display greater structural diversity, complexity, and innovative approaches to problem-solving. Based on these results, we envision the work as groundwork that highlight four agendas to further investigate how GenAI could be utilized in notebooks that maximizes the potential collaboration between human and AI.

Figures (1)

• Figure 1: Overview of Data Collection