Understanding the Helpfulness of Stale Bot for Pull-based Development: An Empirical Study of 20 Large Open-Source Projects

TL;DR

An empirical study of 20 large and popular open source projects finds that Stale bot can help deal with a backlog of unresolved PRs, as the projects closed more PRs within the first few months of adoption.

Abstract

Pull Requests (PRs) that are neither progressed nor resolved clutter the list of PRs, making it difficult for the maintainers to manage and prioritize unresolved PRs. To automatically track, follow up, and close such inactive PRs, Stale bot was introduced by GitHub. Despite its increasing adoption, there are ongoing debates on whether using Stale bot alleviates or exacerbates the problem of inactive PRs. To better understand if and how Stale bot helps projects in their pull-based development workflow, we perform an empirical study of 20 large and popular open-source projects. We find that Stale bot can help deal with a backlog of unresolved PRs as the projects closed more PRs within the first few months of adoption. Moreover, Stale bot can help improve the efficiency of the PR review process as the projects reviewed PRs that ended up merged and resolved PRs that ended up closed faster after the adoption. However, Stale bot can also negatively affect the contributors as the projects experienced a considerable decrease in their number of active contributors after the adoption. Therefore, relying solely on Stale bot to deal with inactive PRs may lead to decreased community engagement and an increased probability of contributor abandonment.

Figures (22)

• Figure 1: An example prompt by Stale bot
• Figure 2: Variation in (a) the number of merged PRs, (b) the number of closed PRs, (c) the first response latency of merged PRs, (d) the resolution time of closed PRs, (e) the number of commits in merged PRs, and (f) the number of active contributors each month during our observation period. The blue lines show the adoption time, the solid red lines show the average predictions, and the dashed red lines show the average counterfactual predictions of the model for all the studied projects. Note that we dropped the outliers in the plots and used the exponential function to convert the log-transformed output of the models back to real values.
• Figure 3: Differences between PRs with and without intervention from Stale bot in the cleverraven/cataclysm-dda project regarding (a) the number of follow-up commits, (b) the number of follow-up changed lines, and (c) the number of follow-up changed files after the submission.
• Figure 4: Differences between the contributors of PRs with and without intervention from Stale bot in the homebrew/homebrew-core project regarding (a) the number of prior PRs, (b) the acceptance rate, and (c) the contribution period.
• Figure 5: Differences between the review processes of PRs with and without intervention from Stale bot in the homebrew/homebrew-core project regarding (a) the number of participants, (b) the number of comments from the participants, (c) the number of comments from the contributors, (d) the first review latency, (e) the mean review latency, and (f) the resolution time.