A Call for Critically Rethinking and Reforming Data Analysis in Empirical Software Engineering

TL;DR

This study argues that three decades of empirical software engineering research harbors widespread statistical and methodological misuses. It combines an automated, LLM-assisted survey of 27,140 studies with a controlled workshop of 33 ESE experts to assess detection and remediation capabilities, revealing consistent gaps in data-analysis rigor. The findings demonstrate poor expert performance in identifying inadequate patterns and proposing correct remedies, underscoring risks of propagating flawed practices through the literature and peer review. The work calls for a systemic reform of data analysis in ESE, emphasizing methodological education, standardized guidelines, and tooling to ensure validity, reproducibility, and industrial relevance. By building a framework for reassessing statistical methodology in ESE, the authors lay groundwork for more reliable and impactful software engineering research.

Abstract

Context: Empirical Software Engineering (ESE) drives innovation in SE through qualitative and quantitative studies. However, concerns about the correct application of empirical methodologies have existed since the 2006 Dagstuhl seminar on SE. Objective: To analyze three decades of SE research, identify mistakes in statistical methods, and evaluate experts' ability to detect and address these issues. Methods: We conducted a literature survey of ~27,000 empirical studies, using LLMs to classify statistical methodologies as adequate or inadequate. Additionally, we selected 30 primary studies and held a workshop with 33 ESE experts to assess their ability to identify and resolve statistical issues. Results: Significant statistical issues were found in the primary studies, and experts showed limited ability to detect and correct these methodological problems, raising concerns about the broader ESE community's proficiency in this area. Conclusions. Despite our study's eventual limitations, its results shed light on recurring issues from promoting information copy-and-paste from past authors' works and the continuous publication of inadequate approaches that promote dubious results and jeopardize the spread of the correct statistical strategies among researchers. Besides, it justifies further investigation into empirical rigor in software engineering to expose these recurring issues and establish a framework for reassessing our field's foundation of statistical methodology application. Therefore, this work calls for critically rethinking and reforming data analysis in empirical software engineering, paving the way for our work soon.

Figures (9)

• Figure 1: Methodological Patterns Example
• Figure 2: Study Setup and Data Collection Process
• Figure 3: Empirical Software Engineering Publication Trend
• Figure 4: Variable Type Trend
• Figure 5: Distribution Check Trend