How Do Software Developers Use GitHub Actions to Automate Their Workflows?

TL;DR

This paper empirically examines how GitHub Actions are adopted and used to automate OSS workflows, evaluating usage patterns, developer discourse, and the impact on pull request dynamics. Using a dataset of 3,190 repositories that adopted Actions and a time-series regression discontinuity design across 926 active projects, the authors quantify Action usage across 708 Actions in 20 categories, categorize discussions from 209 Action-related issues, and measure effects on PRs, comments, latency, and commits. Key findings include a low but growing adoption, mostly positive perception of Actions, and statistically significant changes in commits and PR rejections after adoption, with merged and non-merged PRs showing different sensitivity to adoption. The work provides actionable insights for practitioners considering Action adoption and lays groundwork for further investigation into the upstream effects of automated workflow tools on developer collaboration and productivity.

Abstract

Automated tools are frequently used in social coding repositories to perform repetitive activities that are part of the distributed software development process. Recently, GitHub introduced GitHub Actions, a feature providing automated workflows for repository maintainers. Although several Actions have been built and used by practitioners, relatively little has been done to evaluate them. Understanding and anticipating the effects of adopting such kind of technology is important for planning and management. Our research is the first to investigate how developers use Actions and how several activity indicators change after their adoption. Our results indicate that, although only a small subset of repositories adopted GitHub Actions to date, there is a positive perception of the technology. Our findings also indicate that the adoption of GitHub Actions increases the number of monthly rejected pull requests and decreases the monthly number of commits on merged pull requests. These results are especially relevant for practitioners to understand and prevent undesirable effects on their projects.

Figures (4)

• Figure 1: GitHub workflow automation with GitHub Actions (adapted from GitHub).
• Figure 2: Greetings workflow of Gammapy -- greetings.yml
• Figure 3: Example of github-actions bot greeting a newcomer.
• Figure 4: Number of repositories that use GitHub Actions.