Evolution of statistical analysis in empirical software engineering research: Current state and steps forward

TL;DR

This paper analyzes how statistical analysis is conducted in empirical software engineering (ESE) and why practical significance often lacks explicit treatment. By combining a manual review of 161 papers (2015) with a semi-automatic analysis of 5,196 papers from five top journals (2001–2015), it identifies prevailing practices (strong reliance on parametric tests and effect sizes) and rising but still limited use of distribution tests, power analysis, and nonparametric methods. The authors propose a conceptual statistical analysis workflow linking empirical data, descriptive statistics, hypothesis testing, effect sizes, and practical significance, while advocating Bayesian data analysis and robust reproducibility practices to address current pitfalls. Overall, the work highlights gaps in reporting practical significance, suggests methodological improvements, and provides a road map for more rigorous and context-aware statistical analysis in ESE.

Abstract

Software engineering research is evolving and papers are increasingly based on empirical data from a multitude of sources, using statistical tests to determine if and to what degree empirical evidence supports their hypotheses. To investigate the practices and trends of statistical analysis in empirical software engineering (ESE), this paper presents a review of a large pool of papers from top-ranked software engineering journals. First, we manually reviewed 161 papers and in the second phase of our method, we conducted a more extensive semi-automatic classification of papers spanning the years 2001--2015 and 5,196 papers. Results from both review steps was used to: i) identify and analyze the predominant practices in ESE (e.g., using t-test or ANOVA), as well as relevant trends in usage of specific statistical methods (e.g., nonparametric tests and effect size measures) and, ii) develop a conceptual model for a statistical analysis workflow with suggestions on how to apply different statistical methods as well as guidelines to avoid pitfalls. Lastly, we confirm existing claims that current ESE practices lack a standard to report practical significance of results. We illustrate how practical significance can be discussed in terms of both the statistical analysis and in the practitioner's context.

Figures (8)

• Figure 1: The extraction process, adapted from PetersenFMM2008sysmap, with the parts in bold indicating activities and output that significantly differ from PetersenFMM2008sysmap.
• Figure 2: Overview of the sept tool for semi-automated checking of software engineering research papers. The paper (PDF) file is converted to a text file and is then passed to analyzers that do fuzzy matching (three exemplified here). Each analyzer focuses on finding evidence for (positive), against (negative) or finding no evidence (i.e., skip) for the use of a certain type of statistical test or aspect that is used in the paper. The tool outputs an analysis file that details all evidence found and marks what lead the tool to judge the evidence to be present.
• Figure 3: The mapping of different statistical approaches extracted from papers of the selected journals between 2001--2015.
• Figure 4: The use of distribution tests. One can see that after 2011 the amount of reported tests increases, even though they are still low (e.g., in 2015 all journals combined reached a score of 0.5, when the maximum in the scale is 5).
• Figure 5: The $y$-axis in each chart is the normalization of ratings of the number of papers where we found positive evidence. Notice that the scale on the $y$-axis is still lower than the maximum ratio (5). The thick line is a local regression (loess) of the data.