Oops!... I did it again. Conclusion (In-)Stability in Quantitative Empirical Software Engineering: A Large-Scale Analysis

TL;DR

This paper investigates threats to validity in complex software-repository mining pipelines by performing a large-scale, cross-tool replication across four independent mining tools (Codeface, git2net, GrimoireLab, Kaiaulu) on three influential studies in collaboration, maintenance, and software quality. It combines a lightweight literature review, baseline data comparison, and three full replications to assess data-level discrepancies, their propagation through analyses, and the stability of high-level conclusions. The findings show that while some high-level conclusions can remain robust across tools, substantial tool-induced differences can alter specific results and even conclusions in complex, nuanced research questions. The work emphasizes the need for reproducibility packages and careful tool selection, and it provides concrete recommendations for study design and reporting to improve validity in evolutionary software engineering research.

Abstract

Context: Mining software repositories is a popular means to gain insights into a software project's evolution, monitor project health, support decisions and derive best practices. Tools supporting the mining process are commonly applied by researchers and practitioners, but their limitations and agreement are often not well understood. Objective: This study investigates some threats to validity in complex tool pipelines for evolutionary software analyses and evaluates the tools' agreement in terms of data, study outcomes and conclusions for the same research questions. Method: We conduct a lightweight literature review to select three studies on collaboration and coordination, software maintenance and software quality from high-ranked venues, which we formally replicate with four independent, systematically selected mining tools to quantitatively and qualitatively compare the extracted data, analysis results and conclusions. Results: We find that numerous technical details in tool design and implementation accumulate along the complex mining pipelines and can cause substantial differences in the extracted baseline data, its derivatives, subsequent results of statistical analyses and, under specific circumstances, conclusions. Conclusions: Users must carefully choose tools and evaluate their limitations to assess the scope of validity in an adequate way. Reusing tools is recommended. Researchers and tool authors can promote reusability and help reducing uncertainties by reproduction packages and comparative studies following our approach.

Figures (12)

• Figure 1: Overall study design.
• Figure 2: Informal overview of structural components of the mining tools Codeface, git2net, GrimoireLab and Kaiaulu. White boxes indicate fixed parts of the data extraction pipeline, while coloured boxes visualise configurable steps. Although all tools perform the same analysis steps, their interaction and data structure differ.
• Figure 3: Tree map visualising the popularity of each field identified in the literature review. The size of a square indicates the popularity of the field, measured by the number of studies addressing it. Large italic headings in the upper left corners represent primary fields; bold headings at the bottom of a group squares indicate secondary categories; vertical texts and smallest, lighter coloured squares represent tertiary categories.
• Figure 4: Time series of simple count-based metrics (number of commits, files, developers and finer-grained code entity blocks) calculated based on the git log extracted by Codeface, git2net, GrimoireLab and Kaiaulu with the most similar configurations for the respective replication. Lines are plotted with a small offset to visualise overlapping lines.
• Figure 5: Developer networks constructed by Codeface, git2net, GrimoireLab and Kaiaulu for the same time interval of subject project Flink with the most similar configurations to replicate gote_big_2022. Names have been pseudonymised.