CrashJS: A NodeJS Benchmark for Automated Crash Reproduction
Philip Oliver, Jens Dietrich, Craig Anslow, Michael Homer
TL;DR
CrashJS addresses the lack of JavaScript Automated Crash Reproduction benchmarks by introducing a first-of-its-kind dataset of 453 Node.js crashes collected from BugsJS, GitHub, Syntest-JavaScript, and SecBench.js. It provides a unified crash format, a deduplication pipeline, and novel extraction techniques to support reproducible evaluation of ACR tools, capturing a wide range of crash and program complexities. The analysis reports on crash types, stack-frame counts, and a normalized program-complexity score $C_o$ derived from $F_p$, $F_{max}$, $CCN_p$, and $CCN_{max}$, revealing no strong correlation between crash and program complexity. By enabling standardized comparisons and providing artifacts for reproduction, CrashJS aims to accelerate JavaScript ACR research and the development of robust debugging tools for Node.js ecosystems.
Abstract
Software bugs often lead to software crashes, which cost US companies upwards of $2.08 trillion annually. Automated Crash Reproduction (ACR) aims to generate unit tests that successfully reproduce a crash. The goal of ACR is to aid developers with debugging, providing them with another tool to locate where a bug is in a program. The main approach ACR currently takes is to replicate a stack trace from an error thrown within a program. Currently, ACR has been developed for C, Java, and Python, but there are no tools targeting JavaScript programs. To aid the development of JavaScript ACR tools, we propose CrashJS: a benchmark dataset of 453 Node.js crashes from several sources. CrashJS includes a mix of real-world and synthesised tests, multiple projects, and different levels of complexity for both crashes and target programs.