The Role of Data Filtering in Open Source Software Ranking and Selection

TL;DR

The paper addresses how data filtering in open-source software research biases findings about predictors of popularity. It uses a case study where repository popularity $ns$ is regressed on log-transformed activity measures $ldur$, $lnc$, $lna$, and $lnCore$ via $ns \sim ldur + lnc + lna + lCore$, comparing a control sample to a filtered subset and showing that filtering can flip the sign of the $lnc$ coefficient. The authors discuss missing-data mechanisms (MCAR, MAR, MNAR) and propose stratified sampling on near-complete datasets (World of Code) or weighting strata to preserve representativeness. They argue for using complete data when possible or designing stratified random samples to avoid bias, with implications for reproducibility and generalizability in empirical software engineering.

Abstract

Faced with over 100M open source projects most empirical investigations select a subset. Most research papers in leading venues investigated filtering projects by some measure of popularity with explicit or implicit arguments that unpopular projects are not of interest, may not even represent "real" software projects, or that less popular projects are not worthy of study. However, such filtering may have enormous effects on the results of the studies if and precisely because the sought-out response or prediction is in any way related to the filtering criteria. We exemplify the impact of this practice on research outcomes: how filtering of projects listed on GitHub affects the assessment of their popularity. We randomly sample over 100,000 repositories and use multiple regression to model the number of stars (a proxy for popularity) based on the number of commits, the duration of the project, the number of authors, and the number of core developers. Comparing control with the entire dataset with a filtered model projects having ten or more authors we find that while certain characteristics of the repository consistently predict popularity, the filtering process significantly alters the relation ships between these characteristics and the response. The number of commits exhibited a positive correlation with popularity in the control sample but showed a negative correlation in the filtered sample. These findings highlight the potential biases introduced by data filtering and emphasize the need for careful sample selection in empirical research of mining software repositories. We recommend that empirical work should either analyze complete datasets such as World of Code, or employ stratified random sampling from a complete dataset to ensure that filtering is not biasing the results.