Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JunoBench: A Benchmark Dataset of Crashes in Python Machine Learning Jupyter Notebooks

Yiran Wang, José Antonio Hernández López, Ulf Nilsson, Dániel Varró

TL;DR

JunoBench addresses the lack of crash benchmarks for Python-based ML notebooks by introducing a reproducible dataset of 111 real-world notebook crashes with fixes, drawn from Kaggle and annotated along library, root cause, crash type, and ML pipeline stage. The authors establish a unified Docker-based execution environment and provide reproduced and fixed notebook versions to enable standardized evaluation of debugging tools, crash detection, and diagnosis. They also annotate ground-truth crash-detection and diagnosis labels and demonstrate initial utility with LLM-based crash detection experiments, highlighting both promise and challenges. The dataset enables rigorous benchmarking for notebook-specific bugs, supports reproducible research across ML libraries, and sets the stage for future work on broader library coverage and deployment-time crashes.

Abstract

Jupyter notebooks are widely used for machine learning (ML) prototyping. Yet, few debugging tools are designed for ML code in notebooks, partly, due to the lack of benchmarks. We introduce JunoBench, the first benchmark dataset of real-world crashes in Python-based ML notebooks. JunoBench includes 111 curated and reproducible crashes with verified fixes from public Kaggle notebooks, covering popular ML libraries (e.g., TensorFlow/Keras, PyTorch, Scikit-learn) and notebook-specific out-of-order execution errors. JunoBench ensures reproducibility and ease of use through a unified environment that reliably reproduces all crashes. By providing realistic crashes, their resolutions, richly annotated labels of crash characteristics, and natural-language diagnostic annotations, JunoBench facilitates research on bug detection, localization, diagnosis, and repair in notebook-based ML development.

JunoBench: A Benchmark Dataset of Crashes in Python Machine Learning Jupyter Notebooks

TL;DR

JunoBench addresses the lack of crash benchmarks for Python-based ML notebooks by introducing a reproducible dataset of 111 real-world notebook crashes with fixes, drawn from Kaggle and annotated along library, root cause, crash type, and ML pipeline stage. The authors establish a unified Docker-based execution environment and provide reproduced and fixed notebook versions to enable standardized evaluation of debugging tools, crash detection, and diagnosis. They also annotate ground-truth crash-detection and diagnosis labels and demonstrate initial utility with LLM-based crash detection experiments, highlighting both promise and challenges. The dataset enables rigorous benchmarking for notebook-specific bugs, supports reproducible research across ML libraries, and sets the stage for future work on broader library coverage and deployment-time crashes.

Abstract

Jupyter notebooks are widely used for machine learning (ML) prototyping. Yet, few debugging tools are designed for ML code in notebooks, partly, due to the lack of benchmarks. We introduce JunoBench, the first benchmark dataset of real-world crashes in Python-based ML notebooks. JunoBench includes 111 curated and reproducible crashes with verified fixes from public Kaggle notebooks, covering popular ML libraries (e.g., TensorFlow/Keras, PyTorch, Scikit-learn) and notebook-specific out-of-order execution errors. JunoBench ensures reproducibility and ease of use through a unified environment that reliably reproduces all crashes. By providing realistic crashes, their resolutions, richly annotated labels of crash characteristics, and natural-language diagnostic annotations, JunoBench facilitates research on bug detection, localization, diagnosis, and repair in notebook-based ML development.
Paper Structure (11 sections, 2 figures)

This paper contains 11 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Methodology
  3. Data collection
  4. Benchmark inclusion
  5. Benchmark usability
  6. JunoBench
  7. Potential research opportunities
  8. Evaluating crash detection and repair tools
  9. Initial experiments on LLM-based crash detection
  10. Related work
  11. Conclusion and future work

Figures (2)

  • Figure 1: Overview of the benchmark construction process.
  • Figure 2: Characteristics of JunoBench. Each bar in (b) and (d) is segmented by libraries (a). “TF/K” stands for “TensorFlow/Keras”. “Minor libs” include Statsmodels(2), TorchVision(1), and LightGBM(1).