Quality Gatekeepers: Investigating the Effects ofCode Review Bots on Pull Request Activities
Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, Marco A. Gerosa
TL;DR
The paper tackles how code review bots influence pull request dynamics in GitHub OSS projects, combining a large-scale Regression Discontinuity Design with qualitative interviews to quantify effects and understand their causes. It demonstrates that bot adoption increases merged PRs, decreases non-merged PRs, and generally reduces developer communication, with explanations rooted in enhanced transparency and confidence from bot feedback. The study contributes both methodological insights (RDD in software development) and practical guidance for project members and bot designers, highlighting beneficial effects and potential downsides like information overload or noise. Overall, the findings offer a nuanced view of bot-assisted code review, suggesting careful bot configuration and ongoing evaluation to maximize positive impacts on collaboration and code quality.
Abstract
Software bots have been facilitating several development activities in Open Source Software (OSS) projects, including code review. However, these bots may bring unexpected impacts to group dynamics, as frequently occurs with new technology adoption. Understanding and anticipating such effects is important for planning and management. To analyze these effects, we investigate how several activity indicators change after the adoption of a code review bot. We employed a regression discontinuity design on 1,194 software projects from GitHub. We also interviewed 12 practitioners, including open-source maintainers and contributors. Our results indicate that the adoption of code review bots increases the number of monthly merged pull requests, decreases monthly non-merged pull requests, and decreases communication among developers. From the developers' perspective, these effects are explained by the transparency and confidence the bot comments introduce, in addition to the changes in the discussion focused on pull requests. Practitioners and maintainers may leverage our results to understand, or even predict, bot effects on their projects.