Understanding and Detecting Flaky Builds in GitHub Actions
Wenhao Ge, Chen Zhang
TL;DR
The paper tackles the problem of flaky CI builds in GitHub Actions by conducting a large-scale empirical study using rerun data from 1,960 open-source Java projects. It quantifies reruns, flakiness, and costs, and catalogs 15 failure categories, highlighting that flaky tests, network issues, and dependency resolution are prevalent. To address flaky failures, it introduces FlakeDetector, a machine-learning-based detector that fuses semantic information from build logs with structured CI/CD features, implemented via a two-score fusion and calibrated by forward-chaining validation; it achieves up to a 20.3% improvement in F1 over a strong baseline. The work provides two released datasets and the FlakeDetector tool, offering practical insights to reduce CI waste, increase reliability, and support future research on flaky builds in GitHub Actions.
Abstract
Continuous Integration (CI) is widely used to provide rapid feedback on code changes; however, CI build outcomes are not always reliable. Builds may fail intermittently due to non-deterministic factors, leading to flaky builds that undermine developers' trust in CI, waste computational resources, and threaten the validity of CI-related empirical studies. In this paper, we present a large-scale empirical study of flaky builds in GitHub Actions based on rerun data from 1,960 open-source Java projects. Our results show that 3.2% of builds are rerun, and 67.73% of these rerun builds exhibit flaky behavior, affecting 1,055 (51.28%) of the projects. Through an in-depth failure analysis, we identify 15 distinct categories of flaky failures, among which flaky tests, network issues, and dependency resolution issues are the most prevalent. Building on these findings, we propose a machine learning-based approach for detecting flaky failures at the job level. Compared with a state-of-the-art baseline, our approach improves the F1-score by up to 20.3%.