Automated flakiness detection in quantum software bug reports
Lei Zhang, Andriy Miranskyy
TL;DR
The paper investigates flaky tests in quantum software bug reports, highlighting that quantum nondeterminism and hardware-induced randomness contribute to inconsistent test outcomes. It provides empirical evidence across 14 QC projects, identifying eight flaky-test types and seven fixes, with flakiness accounting for a small but nontrivial fraction of bug reports (0.26%–1.85%). The authors propose framing automated flaky-detection as an ML pipeline, detailing dataset preparation, model development with classical and transfer-learning approaches, validation amidst label noise, and deployment in bug trackers, while addressing data scarcity, imbalance, and drift. This work lays a research agenda for tooling that can automatically flag flaky bugs and guides future efforts to mitigate flakiness as quantum software scales.
Abstract
A flaky test yields inconsistent results upon repetition, posing a significant challenge to software developers. An extensive study of their presence and characteristics has been done in classical computer software but not quantum computer software. In this paper, we outline challenges and potential solutions for the automated detection of flaky tests in bug reports of quantum software. We aim to raise awareness of flakiness in quantum software and encourage the software engineering community to work collaboratively to solve this emerging challenge.