Does the Tool Matter? Exploring Some Causes of Threats to Validity in Mining Software Repositories

TL;DR

Does the Tool Matter? investigates threats to validity in mining software repositories by comparing two MSR tools (Codeface and Kaiaulu) across ten large OSS projects. It quantifies how tool choices, configuration, and post-processing affect baseline VCS data and derived developer networks, finding substantial discrepancies in simple metrics like commits and in edge weights of networks. It shows that some alignment is possible through adjustments but fundamental differences persist, underscoring the need for substantial re-engineering to achieve full interchangeability. The work advocates replication, transparency, and standardization in MSR pipelines to ensure conclusions are robust and generalizable.

Abstract

Software repositories are an essential source of information for software engineering research on topics such as project evolution and developer collaboration. Appropriate mining tools and analysis pipelines are therefore an indispensable precondition for many research activities. Ideally, valid results should not depend on technical details of data collection and processing. It is, however, widely acknowledged that mining pipelines are complex, with a multitude of implementation decisions made by tool authors based on their interests and assumptions. This raises the questions if (and to what extent) tools agree on their results and are interchangeable. In this study, we use two tools to extract and analyse ten large software projects, quantitatively and qualitatively comparing results and derived data to better understand this concern. We analyse discrepancies from a technical point of view, and adjust code and parametrisation to minimise replication differences. Our results indicate that despite similar trends, even simple metrics such as the numbers of commits and developers may differ by up to 500%. We find that such substantial differences are often caused by minor technical details. We show how tool-level and data post-processing changes can overcome these issues, but find they may require considerable efforts. We summarise identified causes in our lessons learned to help researchers and practitioners avoid common pitfalls, and reflect on implementation decisions and their influence in ensuring obtained data meets explicit and implicit expectations. Our findings lead us to hypothesise that similar uncertainties exist in other analysis tools, which may limit the validity of conclusions drawn in tool-centric research.

Figures (5)

• Figure 1: Overview about our approach: We mined repositories using a basis (Codeface) and replication (Kaiaulu) tool, and compared derived data. Depending on similarities and differences, we adjusted configuration parameters and post-processing methods in the replication tool. We iterated until the closest replication was achieved.
• Figure 2: Informal overview of structural components of the mining tools Codeface and Kaiaulu. Although both tools perform the same analysis steps, their interaction and data structure differ.
• Figure 3: Time series of count-based metrics calculated based on the git log extracted by Codeface and Kaiaulu with prior (P) configuration and Kaiaulu with replication (R) configuration. Lines are plotted with a small horizontal offset to visualise overlapping lines. We see similar trends for both tools, but notice that the extent of discrepancies is project- and configuration-specific. The most prominent differences can be observed for project Spark.
• Figure 4: Jointly identified files, entities and developers, considering prior (P) and replication (R) configurations over the entire VCS history. Again, we observe that the magnitude of discrepancies depends on the project, the considered metric and the tool configuration.
• Figure 5: Developer networks constructed by Codeface (orange) and Kaiaulu with prior configuration (blue) for Spark from March--June '13. Two very active developers missed by Kaiaulu are marked red. The discrepancies in networks are due to Kaiaulu not supporting the Scala language.