Predicting the First Response Latency of Maintainers and Contributors in Pull Requests

TL;DR

This study tackles the practical problem of predicting the first response latency in PR reviews for both maintainers and contributors. It builds a dataset from 20 large GitHub projects, extracting 21 features across project, contributor, PR, and review-process dimensions, and trains seven classifiers, with CatBoost consistently delivering the best performance across project-specific and cross-project settings. The authors quantify improvements over a majority-baseline using $AUC$-ROC and $AUC$-PR and analyze feature importance with permutation tests and SHAP to identify top predictors, such as submission timing, PR size, and contributor history. The findings offer concrete guidelines for contributors and maintainers to manage expectations and improve collaboration, and demonstrate that cross-project models can generalize to help new projects anticipate PR review dynamics.

Abstract

The success of a Pull Request (PR) depends on the responsiveness of the maintainers and the contributor during the review process. Being aware of the expected waiting times can lead to better interactions and managed expectations for both the maintainers and the contributor. In this paper, we propose a machine-learning approach to predict the first response latency of the maintainers following the submission of a PR, and the first response latency of the contributor after receiving the first response from the maintainers. We curate a dataset of 20 large and popular open-source projects on GitHub and extract 21 features to characterize projects, contributors, PRs, and review processes. Using these features, we then evaluate seven types of classifiers to identify the best-performing models. We also conduct permutation feature importance and SHAP analyses to understand the importance and the impact of different features on the predicted response latencies. We find that our CatBoost models are the most effective for predicting the first response latencies of both maintainers and contributors. We also observe that PRs submitted earlier in the week, containing an average number of commits, and with concise descriptions are more likely to receive faster first responses from the maintainers. Similarly, PRs with a lower first response latency from maintainers, that received the first response of maintainers earlier in the week, and containing an average number of commits tend to receive faster first responses from the contributors. Additionally, contributors with a higher acceptance rate and a history of timely responses in the project are likely to both obtain and provide faster first responses. Moreover, we show the effectiveness of our approach in a cross-project setting.

Figures (6)

• Figure 1: Ranking of the importance of different features for predicting the first response latency of maintainers across the studied projects. Darker colors indicate higher importance.
• Figure 2: Impact of the top 5 most important features on the prediction of the first response latency of maintainers across the studied projects. Wider violins indicate higher density and more frequent values.
• Figure 3: Ranking of the importance of different features for predicting the first response latency of contributors across the studied projects. Darker colors indicate higher importance.
• Figure 4: Impact of the top 5 most important features on the prediction of the first response latency of contributors across the studied projects. Wider violins indicate higher density and more frequent values.
• Figure 5: Ranking of the importance of different features for predicting the first response latency of maintainers in a cross-project scenario.