GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions
Nuno Saavedra, André Silva, Martin Monperrus
TL;DR
This work tackles the need for up-to-date, fully reproducible bug-fix benchmarks for evaluating software engineering techniques such as automatic program repair and fault localization. It introduces GitBug-Actions, a GitHub Actions–based workflow that identifies recent bug fixes and reconstructs fully reproducible offline environments to ensure long-term validity. The approach is validated with a Go PoC, yielding 21 fully reproducible bug-fix commits from 13 repositories, illustrating the workflow's practicality and scalability across languages. The contribution is a public, language-agnostic framework for collecting robust bug-fix benchmarks that reflect modern development practices and CI usage, enabling more reliable benchmarking over time.
Abstract
Bug-fix benchmarks are fundamental in advancing various sub-fields of software engineering such as automatic program repair (APR) and fault localization (FL). A good benchmark must include recent examples that accurately reflect technologies and development practices of today. To be executable in the long term, a benchmark must feature test suites that do not degrade overtime due to, for example, dependencies that are no longer available. Existing benchmarks fail in meeting both criteria. For instance, Defects4J, one of the foremost Java benchmarks, last received an update in 2020. Moreover, full-reproducibility has been neglected by the majority of existing benchmarks. In this paper, we present GitBug-Actions: a novel tool for building bug-fix benchmarks with modern and fully-reproducible bug-fixes. GitBug-Actions relies on the most popular CI platform, GitHub Actions, to detect bug-fixes and smartly locally execute the CI pipeline in a controlled and reproducible environment. To the best of our knowledge, we are the first to rely on GitHub Actions to collect bug-fixes. To demonstrate our toolchain, we deploy GitBug-Actions to build a proof-of-concept Go bug-fix benchmark containing executable, fully-reproducible bug-fixes from different repositories. A video demonstrating GitBug-Actions is available at: https://youtu.be/aBWwa1sJYBs.