Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis

TL;DR

<3-5 sentence high-level summary> The paper addresses the fragility of empirical software engineering findings arising from researcher degrees of freedom in data analysis. It employs a multiverse analysis on a published Mining Software Repositories study (RDiT) to systematically vary nine analytical decisions, yielding 3,072 plausible pipelines and 12,288 models; only 6 universes reproduce the original results, and many yield contradictory or inconclusive outcomes. The authors argue for robustness checks and introduce the Justification Ladder of Analytical Choices (JLAC) to elevate justification for analytical decisions, along with practical guidelines and time-series-specific considerations for RDiT. The work highlights the need for greater transparency and methodological rigor in MSR research and proposes multiverse analyses as a diagnostic tool to improve reliability and reproducibility in empirical software engineering.

Abstract

In empirical software engineering (SE) research, researchers have considerable freedom to decide how to process data, what operationalizations to use, and which statistical model to fit. Gelman and Loken refer to this freedom as leading to a "garden of forking paths". Although this freedom is often seen as an advantage, it also poses a threat to robustness and replicability: variations in analytical decisions, even when justifiable, can lead to divergent conclusions. To better understand this risk, we conducted a so-called multiverse analysis on a published empirical SE paper. The paper we picked is a Mining Software Repositories study, as MSR studies commonly use non-trivial statistical models to analyze post-hoc, observational data. In the study, we identified nine pivotal analytical decisions-each with at least one equally defensible alternative and systematically reran all the 3,072 resulting analysis pipelines on the original dataset. Interestingly, only 6 of these universes (<0.2%) reproduced the published results; the overwhelming majority produced qualitatively different, and sometimes even opposite, findings. This case study of a data analytical method commonly applied to empirical software engineering data reveals how methodological choices can exert a more profound influence on outcomes than is often acknowledged. We therefore advocate that SE researchers complement standard reporting with robustness checks across plausible analysis variants or, at least, explicitly justify each analytical decision. We propose a structured classification model to help classify and improve justification for methodological choices. Secondly, we show how the multiverse analysis is a practical tool in the methodological arsenal of SE researchers, one that can help produce more reliable, reproducible science.

Figures (6)

• Figure 1: Visualization of the core idea of a multiverse analysis. A study moves from research question (blue node) to outcome (green node) through a path of methodological decisions (black nodes). In a multiverse analysis, alternative paths (dashed arrows) are explored to study whether these alternatives change outcomes (red node).
• Figure 2: An example of Regression Discontinuity in Time (RDiT) design. The plot shows two scenarios; in both scenarios, there is an intervention taking place in the 12th month. The left plot shows a scenario where the intervention does not have any effect, with no change in the outcome after the intervention, while the right plot illustrates a scenario where the intervention results in an observable change, with a visible discontinuity both in slope and intercept of the outcome.
• Figure 3: High-level overview of the outcomes for all of the multiverses, per dependent variable. TODO$\blacktriangleright$Highlight this as a red flag?$\blacktriangleleft$
• Figure 4: A specification curve showing the number of hypotheses that can be confirmed in each universe, and the relation between a universe, and the decisions made within in the universe.
• Figure 5: Distribution plots showing the percentage of universes in which changing one decision leads to an alternative outcome.