Does the First Response Matter for Future Contributions? A Study of First Contributions

TL;DR

This study addresses whether the first response to a newcomer's pull request influences their likelihood of future contributions in OSS. Using a large-scale, registered-report-inspired protocol on $2{,}765{,}917$ PRs and $642{,}841$ first responses from $72{,}635$ projects, it characterizes first responses across sentiment, responsiveness, toxicity, and emotion, and investigates their impact on future interactions via both quantitative and qualitative analyses. The key findings show first contributions receive more positive sentiment but are less responsive, toxicity is not biased toward first contributions, and emotions such as fear, joy, and love are more prevalent in first responses; positive or fast first responses do not reliably increase future interactions, while negative responses tend to be constructive or critical. A predictive ML framework (Random Forest) achieves AUROC around $0.6671$ and F1 around $0.6171$, with feature importance dominated by project-, contributor-, and contribution-level attributes rather than sentiment alone, suggesting that future participation hinges more on structural and contextual factors. Overall, the work provides nuanced guidance for OSS communities: constructive, non-toxic feedback is acceptable and may support onboarding, but interventions to boost ongoing contributions should consider broader project and contributor dynamics beyond first-response sentiment. Replication data and datasets are publicly available to support further analysis and replication.

Abstract

Open Source Software (OSS) projects rely on a continuous stream of new contributors for their livelihood. Recent studies reported that new contributors experience many barriers in their first contribution, with the social barrier being critical. Although a number of studies investigated the social barriers to new contributors, we hypothesize that negative first responses may cause an unpleasant feeling, and subsequently lead to the discontinuity of any future contribution. We execute protocols of a registered report to analyze 2,765,917 first contributions as Pull Requests (PRs) with 642,841 first responses. We characterize most first response as being positive, but less responsive, and exhibiting sentiments of fear, joy and love. Results also indicate that negative first responses have the literal intention to arouse emotions of being either constructive (50.71%) or criticizing (37.68%) in nature. Running different machine learning models, we find that predicting future interactions is low (F1 score of 0.6171), but relatively better than baselines. Furthermore, an analysis of these models show that interactions are positively correlated with a future contribution, with other dimensions (i.e., project, contributor, contribution) having a large effect.

Figures (6)

• Figure 1: A screenshot example of a negative response to a new contributor, which may deter them interacting in the future.
• Figure 2: Data Preparation Overview
• Figure 3: Visual analysis of Sentiment Score (a) and Responsiveness (b) of the First Response
• Figure 4: Quantitative results to answer RQ2
• Figure 5: Feature Permutation Importance for RQ3