Social Proof is in the Pudding: The (Non)-Impact of Social Proof on Software Downloads

TL;DR

A study of the impact of buy'stars for a random set of GitHub repositories of new Python packages on the largest developer platform, GitHub, which finds no detectable effect on subsequent downloads or on any measure of repository activity.

Abstract

Open-source software is widely used in commercial applications. Pair that with the fact that when choosing open-source software for a new problem, developers often use social proof as a cue. These two facts raise concerns that bad actors can game social proof metrics to induce the use of malign software. We study the question using two field experiments. On the largest developer platform, GitHub, we buy 'stars' for a random set of GitHub repositories of new Python packages and estimate their impact on package downloads and broader repository activity. We find no discernible impact on downloads, nor on forks, pull requests, issues, or other measures of developer engagement. In another field experiment, we manipulate the number of human downloads for Python packages. Again, we find no detectable effect on subsequent downloads or on any measure of repository activity.

Figures (9)

• Figure 1: Manipulation Check: GitHub Stars for Treated vs. Control. The figure plots the median number of stars on a particular day for the three groups: high-dosage treatment (market and network stars), low-dosage treatment (network stars), and control (no stars). In all, we plot data for 585 packages and 17,550 package days. The shaded vertical bar indicates the period during which the treatment was applied. \ref{['tab:github_exp_stars_regtable']} presents formal estimates of this manipulation check. See \ref{['fig:timeseries-stars-treated-012']} for the figure that shows the means.
• Figure 2: Median PyPI Downloads for Treatment vs. Control in GitHub Experiment. The figure plots the median of cumulative downloads for each of the three groups. Each point is a day averaged within the group for 622 packages and 118,180 package days. Treatment is distinguished by low and high dosage (see \ref{['sec:gh-design']}). The shaded vertical bar indicates the treatment period. Downloads include only human downloads (\ref{['tab:human_bot_download_classification']}). See also fig:timeseries-downloads-treated-012-individualfig:timeseries-downloads-treated-012-without-top2-extreme-outliers for the time series of individual packages. \ref{['tab:github_exp_medians_regtable']} reports estimates of differences in medians. \ref{['fig:timeseries-downloads-treated-012']} plots the means.
• Figure 3: Median PyPI downloads for Treated vs. Control in the PyPI Experiment. The figure shows trends in median daily downloads for the treated packages (n = 4,814) and control group packages (n = 19,102) for 1,458,876 package-day observations. The shaded vertical bar indicates the treatment period. Downloads include only human downloads (\ref{['tab:human_bot_download_classification']}). See \ref{['fig:mean_downloads_pypi_experiment']} for the same figure of mean downloads. \ref{['tab:pypi_exp_regtable']} reports the estimates for the differences in medians.
• Figure SI 1.1: Manipulation Check: GitHub Stars for Treated vs. Control. Same as \ref{['fig:timeseries-stars-treated-012-medians']}, except in means. The figure reports the means version of \ref{['fig:timeseries-stars-treated-012-medians']}. The figure plots the mean number of stars on a particular day for the three groups: high-dosage treatment (market and network stars), low-dosage treatment (market stars), and control (no stars). In all, we plot data for 585 packages and 17,550 package days. The shaded vertical bar indicates the period during which the treatment was being applied. \ref{['tab:github_exp_stars_regtable']} presents estimates of the impact of treatment on downloads.
• Figure SI 1.2: Mean PyPI downloads for Treated vs. Control in GitHub Experiment. Same as \ref{['fig:timeseries-downloads-treated-012-medians']}, except for means. The figure plots the mean of cumulative downloads for each of the three groups. Each point is a day averaged within the group for 622 packages and 118,180 package days. Treatment is distinguished by low and high dosages (see \ref{['sec:gh-design']}). The shaded vertical bar indicates the treatment period. Downloads include only human downloads (\ref{['tab:human_bot_download_classification']}). See also fig:timeseries-downloads-treated-012-individualfig:timeseries-downloads-treated-012-without-top2-extreme-outliers for the time series of individual packages. \ref{['tab:github_exp_regtable']} reports the estimates of differences in means.