How do Software Engineering Researchers Use GitHub? An Empirical Study of Artifacts & Impact

TL;DR

This study analyzes how software engineering researchers publish code on GitHub alongside publications, using a data-driven pipeline over about 10k papers and 3,449 GitHub links to study 309 author-owned repositories. It reveals a wide variation in repository popularity and maintenance, with many projects receiving little or no public interaction while a few achieve substantial impact and correlate with higher citations. Author responsiveness to issues is often slow or absent, suggesting that social coding expectations are not routinely met in academia. The findings argue for rethinking incentives around publishing and maintaining research artifacts, emphasizing better documentation, archiving, and recognition of maintenance as part of scholarly impact.

Abstract

Millions of developers share their code on open-source platforms like GitHub, which offer social coding opportunities such as distributed collaboration and popularity-based ranking. Software engineering researchers have joined in as well, hosting their research artifacts (tools, replication package & datasets) in repositories, an action often marked as part of the publications contribution. Yet a decade after the first such paper-with-GitHub-link, little is known about the fate of such repositories in practice. Do research repositories ever gain the interest of the developer community, or other researchers? If so, how often and why (not)? Does effort invested on GitHub pay off with research impact? In short: we ask whether and how authors engage in social coding related to their research. We conduct a broad empirical investigation of repositories from published work, starting with ten thousand papers in top SE research venues, hand-annotating their 3449 GitHub (and Zenodo) links, and studying 309 paper-related repositories in detail. We find a wide distribution in popularity and impact, some strongly correlated with publication venue. These were often heavily informed by the authors investment in terms of timely responsiveness and upkeep, which was often remarkably subpar by GitHubs standards, if not absent altogether. Yet we also offer hope: popular repositories often go hand-in-hand with well-citepd papers and achieve broad impact. Our findings suggest the need to rethink the research incentives and reward structure around research products requiring such sustained contributions.

Figures (7)

• Figure 1: Our data collection pipeline. We first collect ca. 20K papers from selected venues (1) and ca. 3.4K GitHub links from these (2), which we manually annotate (3) to inform RQ1. We then gather the "social" statistics of author-published repositories (4) for RQ2, and finally study some of these, especially more remarkable, repositories (and their papers) in-depth to gain richer insights for RQ3.
• Figure 2: Distribution and change of GitHub links in selected publications over time.
• Figure 3: The top $10$ programming languages used in author-owned repositories.
• Figure 4: Adoption of research-related GitHub repositories by venue over time. Note that recent (esp. 2020) tallies are partial, as papers and repositories may not yet be public.
• Figure 5: Distribution of the number of common types of public interactions by year across all academic repositories on GitHub (?: Closed ?: Open).