Apples, Oranges, and Software Engineering: Study Selection Challenges for Secondary Research on Latent Variables

TL;DR

The paper investigates how latent variables in software engineering pose study-selection challenges for secondary research due to inconsistent measurement and definitions. It presents two case studies—a meta-analysis attempt and a systematic mapping study—to illustrate how heterogeneity impedes synthesis. Drawing from psychometrics and medicine, it argues for explicit construct definitions and validated measurement instruments, complemented by open-data practices to support replication and instrument validation. The proposed two-step forward—define/model/discuss constructs and standardize/validate instruments—offers a practical roadmap to make SE secondary studies more rigorous, reproducible, and actionable for researchers and practitioners.

Abstract

Software engineering (SE) is full of abstract concepts that are crucial for both researchers and practitioners, such as programming experience, team productivity, code comprehension, and system security. Secondary studies aimed at summarizing research on the influences and consequences of such concepts would therefore be of great value. However, the inability to measure abstract concepts directly poses a challenge for secondary studies: primary studies in SE can operationalize such concepts in many ways. Standardized measurement instruments are rarely available, and even if they are, many researchers do not use them or do not even provide a definition for the studied concept. SE researchers conducting secondary studies therefore have to decide a) which primary studies intended to measure the same construct, and b) how to compare and aggregate vastly different measurements for the same construct. In this experience report, we discuss the challenge of study selection in SE secondary research on latent variables. We report on two instances where we found it particularly challenging to decide which primary studies should be included for comparison and synthesis, so as not to end up comparing apples with oranges. Our report aims to spark a conversation about developing strategies to address this issue systematically and pave the way for more efficient and rigorous secondary studies in software engineering.