Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Runtime-Augmented LLMs for Crash Detection and Diagnosis in ML Notebooks

Yiran Wang, José Antonio Hernández López, Ulf Nilsson, Dániel Varró

TL;DR

CRANE-LLM is presented, a novel approach that augments large language models (LLMs) with structured runtime information extracted from the notebook kernel state to detect and diagnose crashes before executing a target cell.

Abstract

Jupyter notebooks are widely used for machine learning (ML) development due to their support for interactive and iterative experimentation. However, ML notebooks are highly prone to bugs, with crashes being among the most disruptive. Despite their practical importance, systematic methods for crash detection and diagnosis in ML notebooks remain largely unexplored. We present CRANE-LLM, a novel approach that augments large language models (LLMs) with structured runtime information extracted from the notebook kernel state to detect and diagnose crashes before executing a target cell. Given previously executed cells and a target cell, CRANE-LLM combines static code context with runtime information, including object types, tensor shapes, and data attributes, to predict whether the target cell will crash (detection) and explain the underlying cause (diagnosis). We evaluate CRANE-LLM on JunoBench, a benchmark of 222 ML notebooks comprising 111 pairs of crashing and corresponding non-crashing notebooks across multiple ML libraries and crash root causes. Across three state-of-the-art LLMs (Gemini, Qwen, and GPT-5), runtime information improves crash detection and diagnosis by 7-10 percentage points in accuracy and 8-11 in F1-score, with larger gains for diagnosis. Improvements vary across ML libraries, crash causes, and LLMs, and depends on the integration of complementary categories of runtime information.

Runtime-Augmented LLMs for Crash Detection and Diagnosis in ML Notebooks

TL;DR

CRANE-LLM is presented, a novel approach that augments large language models (LLMs) with structured runtime information extracted from the notebook kernel state to detect and diagnose crashes before executing a target cell.

Abstract

Jupyter notebooks are widely used for machine learning (ML) development due to their support for interactive and iterative experimentation. However, ML notebooks are highly prone to bugs, with crashes being among the most disruptive. Despite their practical importance, systematic methods for crash detection and diagnosis in ML notebooks remain largely unexplored. We present CRANE-LLM, a novel approach that augments large language models (LLMs) with structured runtime information extracted from the notebook kernel state to detect and diagnose crashes before executing a target cell. Given previously executed cells and a target cell, CRANE-LLM combines static code context with runtime information, including object types, tensor shapes, and data attributes, to predict whether the target cell will crash (detection) and explain the underlying cause (diagnosis). We evaluate CRANE-LLM on JunoBench, a benchmark of 222 ML notebooks comprising 111 pairs of crashing and corresponding non-crashing notebooks across multiple ML libraries and crash root causes. Across three state-of-the-art LLMs (Gemini, Qwen, and GPT-5), runtime information improves crash detection and diagnosis by 7-10 percentage points in accuracy and 8-11 in F1-score, with larger gains for diagnosis. Improvements vary across ML libraries, crash causes, and LLMs, and depends on the integration of complementary categories of runtime information.
Paper Structure (51 sections, 3 equations, 5 figures, 5 tables)

This paper contains 51 sections, 3 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. ML notebooks
  4. Running example
  5. Runtime information in ML notebooks
  6. Kernel state corruption in notebooks
  7. The CRANE-LLM Approach
  8. Runtime information extraction
  9. Tensors, arrays, and dataframes
  10. ML library-specific data
  11. Example
  12. Crash detection and diagnosis
  13. Example
  14. CRANE-LLM configurations
  15. Runtime information
  16. ...and 36 more sections

Figures (5)

  • Figure 1: A JunoBench notebook example showing the notebook structure (left), kernel state (middle), and showcase the CRANE-LLM pipeline.
  • Figure 2: A simulated notebook example showing how a crashing cell can irreversibly corrupt the kernel state by updating model parameters before the crash occurs.
  • Figure 3: Overview of CRANE-LLM.
  • Figure 4: Overview of experimental setup.
  • Figure 5: Runtime information improvements on JunoBench.