Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Hidden Costs of Automation: An Empirical Study on GitHub Actions Workflow Maintenance

Pablo Valenzuela-Toledo, Alexandre Bergel, Timo Kehrer, Oscar Nierstrasz

TL;DR

This study addresses the maintenance costs of GitHub Actions workflows by conducting a large-scale empirical analysis across 183 GA-enabled projects in 10 languages. Using a mix of statistical tests, association-rule mining, and qualitative coding, it characterizes the volume, evolution, coupling, and ownership of GA workflow files. Key findings show that workflows are a small but sizable code artifact with relatively low churn, weak but meaningful coupling to production and test code, and a pattern of specialized yet cross-functional developer involvement. The work provides actionable guidance for resource planning, best-practice documentation, and tooling improvements to better manage GA workflow maintenance and dependency tracking.

Abstract

GitHub Actions (GA) is an orchestration platform that streamlines the automatic execution of software engineering tasks such as building, testing, and deployment. Although GA workflows are the primary means for automation, according to our experience and observations, human intervention is necessary to correct defects, update dependencies, or refactor existing workflow files. In fact, previous research has shown that software artifacts similar to workflows, such as build files and bots, can introduce additional maintenance tasks in software projects. This suggests that workflow files, which are also used to automate repetitive tasks in professional software production, may generate extra workload for developers. However, the nature of such effort has not been well studied. This paper presents a large-scale empirical investigation towards characterizing the maintenance of GA workflows by studying the evolution of workflow files in almost 200 mature GitHub projects across ten programming languages. Our findings largely confirm the results of previous studies on the maintenance of similar artifacts, while also revealing GA-specific insights such as bug fixing and CI/CD improvement being among the major drivers of GA maintenance. A direct implication is that practitioners should be aware of proper resource planning and allocation for maintaining GA workflows, thus exposing the ``hidden costs of automation.'' Our findings also call for identifying and documenting best practices for such maintenance, and for enhanced tool features supporting dependency tracking and better error reporting of workflow specifications.

The Hidden Costs of Automation: An Empirical Study on GitHub Actions Workflow Maintenance

TL;DR

This study addresses the maintenance costs of GitHub Actions workflows by conducting a large-scale empirical analysis across 183 GA-enabled projects in 10 languages. Using a mix of statistical tests, association-rule mining, and qualitative coding, it characterizes the volume, evolution, coupling, and ownership of GA workflow files. Key findings show that workflows are a small but sizable code artifact with relatively low churn, weak but meaningful coupling to production and test code, and a pattern of specialized yet cross-functional developer involvement. The work provides actionable guidance for resource planning, best-practice documentation, and tooling improvements to better manage GA workflow maintenance and dependency tracking.

Abstract

GitHub Actions (GA) is an orchestration platform that streamlines the automatic execution of software engineering tasks such as building, testing, and deployment. Although GA workflows are the primary means for automation, according to our experience and observations, human intervention is necessary to correct defects, update dependencies, or refactor existing workflow files. In fact, previous research has shown that software artifacts similar to workflows, such as build files and bots, can introduce additional maintenance tasks in software projects. This suggests that workflow files, which are also used to automate repetitive tasks in professional software production, may generate extra workload for developers. However, the nature of such effort has not been well studied. This paper presents a large-scale empirical investigation towards characterizing the maintenance of GA workflows by studying the evolution of workflow files in almost 200 mature GitHub projects across ten programming languages. Our findings largely confirm the results of previous studies on the maintenance of similar artifacts, while also revealing GA-specific insights such as bug fixing and CI/CD improvement being among the major drivers of GA maintenance. A direct implication is that practitioners should be aware of proper resource planning and allocation for maintaining GA workflows, thus exposing the ``hidden costs of automation.'' Our findings also call for identifying and documenting best practices for such maintenance, and for enhanced tool features supporting dependency tracking and better error reporting of workflow specifications.
Paper Structure (23 sections, 3 figures, 4 tables)

This paper contains 23 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Research Questions
  3. Study Design
  4. Project Selection
  5. Data Curation
  6. Analysis Techniques
  7. Analysis and Results
  8. Relative Frequency and Size of Workflow Files (RQ1)
  9. Rate of Changes in Workflow Files (RQ2)
  10. Quantitative Analysis of Logical Workflow Coupling (RQ3)
  11. Qualitative Analysis of Logical Workflow Coupling (RQ4)
  12. Management and Configuration
  13. Development and Enhancement
  14. Bug Fixes and Compatibility
  15. Testing and Documentation
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Mean relative frequencies of the studied file categories in the multi-workflow group.
  • Figure 2: Mean file sizes in NLOC (without outliers) of the studied file categories in the multi-workflow group.
  • Figure 3: Distribution of the rate of changes for the workflow, test, and production file categories. We calculated the monthly rate of change by normalizing the number of changed files against the total files in each category, then averaging this proportion per month for each project.