Teaching Software Metrology: The Science of Measurement for Software Engineering

TL;DR

This chapter reviews key concepts in the science of measurement and applies them to software engineering research and a series of exercises for applying important measurement concepts to the reader's research are included.

Abstract

While the methodological rigor of computing research has improved considerably in the past two decades, quantitative software engineering research is hampered by immature measures and inattention to theory. Measurement-the principled assignment of numbers to phenomena-is intrinsically difficult because observation is predicated upon not only theoretical concepts but also the values and perspective of the research. Despite several previous attempts to raise awareness of more sophisticated approaches to measurement and the importance of quantitatively assessing reliability and validity, measurement issues continue to be widely ignored. The reasons are unknown, but differences in typical engineering and computer science graduate training programs (compared to psychology and management, for example) are involved. This chapter therefore reviews key concepts in the science of measurement and applies them to software engineering research. A series of exercises for applying important measurement concepts to the reader's research are included, and a sample dataset for the reader to try some of the statistical procedures mentioned is provided.

Figures (10)

• Figure 1: Simple example of causal chain
• Figure 2: Examples of Abstracting to a Broader Construct of Interest
• Figure 3: Sufficient (a) vs. Insufficient (b) reliability given expected effect size of super shoes (blue) over regular shoes (red)
• Figure 4: Systematic Error in Heart Rate Monitor. True heart rate (blue) vs. Measured heart rate (red), underestimating by 10 BPM
• Figure 5: Hypothesized Latent Structure