Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Test Code Review in the Era of GitHub Actions: A Replication Study

Hui Sun, Yinan Wu, Wesley K. G. Assunção, Kathryn T. Stolee

Abstract

Test code is indispensable in software development, ensuring the correctness of production code and supporting maintainability. Nonetheless, errors or omissions in the test code can conceal production defects. While code review is widely adopted to assess code quality and correctness, little research has examined how test code is reviewed. Spadini et al.'s research on Gerrit (a pre-commit review model) found that test code receives significantly less discussion than production code. However, the most popular review model is currently based on pull requests (PRs), in which contributors propose changes for discussion and approval, a more negotiable and flexible model compared to Gerrit. Furthermore, GitHub Actions (GHA) has become widely used to automate pre-checks and testing, potentially impacting review practices. This leads us to explore whether Spadini et al.'s findings still hold for the PR model in the era of GHA? Our work replicates and extends their work. We focus on GitHub PRs and analyze six open-source projects. We investigate the impact of the PR model and GHA on test code review. Our results show that GitHub's PR model fosters more balanced discussions between test and production files than Gerrit, albeit with lower overall comment density. However, despite cross-project heterogeneity, GHA adoption triggered a sharp pivot toward production code. Post-GHA, for PRs involving tests, both review probability and comment density reached a median of zero. These findings reveal how evolving continuous integration pipelines can marginalize test code review. The observed decline in test-centric discussion under GHA warrants concern regarding long-term software quality. Our work also presents recommendations for stakeholders involved in the software development life cycle.

Test Code Review in the Era of GitHub Actions: A Replication Study

Abstract

Test code is indispensable in software development, ensuring the correctness of production code and supporting maintainability. Nonetheless, errors or omissions in the test code can conceal production defects. While code review is widely adopted to assess code quality and correctness, little research has examined how test code is reviewed. Spadini et al.'s research on Gerrit (a pre-commit review model) found that test code receives significantly less discussion than production code. However, the most popular review model is currently based on pull requests (PRs), in which contributors propose changes for discussion and approval, a more negotiable and flexible model compared to Gerrit. Furthermore, GitHub Actions (GHA) has become widely used to automate pre-checks and testing, potentially impacting review practices. This leads us to explore whether Spadini et al.'s findings still hold for the PR model in the era of GHA? Our work replicates and extends their work. We focus on GitHub PRs and analyze six open-source projects. We investigate the impact of the PR model and GHA on test code review. Our results show that GitHub's PR model fosters more balanced discussions between test and production files than Gerrit, albeit with lower overall comment density. However, despite cross-project heterogeneity, GHA adoption triggered a sharp pivot toward production code. Post-GHA, for PRs involving tests, both review probability and comment density reached a median of zero. These findings reveal how evolving continuous integration pipelines can marginalize test code review. The observed decline in test-centric discussion under GHA warrants concern regarding long-term software quality. Our work also presents recommendations for stakeholders involved in the software development life cycle.
Paper Structure (28 sections, 2 equations, 6 figures, 3 tables)

This paper contains 28 sections, 2 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Original Study
  3. Replication Scope and Methodology
  4. Research Questions
  5. Data Processing
  6. Project Selection
  7. Data Cleaning
  8. Evaluation Metrics
  9. RQ1 Considerations
  10. RQ2 Considerations
  11. RQ3 Considerations
  12. Result
  13. RQ1: How is reviewers’ attention distributed between production and test code, and does GHA alter this distribution?
  14. Platform Level Comparison: Gerrit vs. GitHub
  15. GHA Influence: Odds Ratio Perspective
  16. ...and 13 more sections

Figures (6)

  • Figure 1: Overview of the Time-based RDD Methodology.
  • Figure 2: Analysis of code review prevalence and GHA impact. (a) Highlights the variance across projects, while (b) illustrates the correlation with GHA adoption.
  • Figure 3: Interrupted Time Series Analysis (ITSA) of test file review metrics before and after GHA adoption. (a) shows the trend in test code review rates, while (b) illustrates the change in test code review density.
  • Figure 4: Distribution and categories of topics identified in the manual analysis of test code reviews ($N=770$). The classification highlights prevalent concerns raised by reviewers during the assessment of test suites.
  • Figure 5: GHA influence on Reviewer Review Preference
  • ...and 1 more figures