On (Mis)perceptions of testing effectiveness: an empirical study

TL;DR

This work investigates whether developers’ perceptions of defect-detection technique effectiveness reflect reality in the absence of prior experience. Through an original controlled experiment and a replication with 46 students, the authors show that perceptions are often wrong and incur a measurable cost to software quality. The replicated study adds participants’ opinions to probe drivers behind misperceptions, finding that opinions do not reliably predict actual effectiveness and that perceived effectiveness is linked to how well techniques are applied. The results motivate the design of feedback tools and broader empirical work to reduce reliance on biased perceptions when selecting defect-detection strategies, with practical implications for practice and research.

Abstract

A recurring problem in software development is incorrect decision making on the techniques, methods and tools to be used. Mostly, these decisions are based on developers' perceptions about them. A factor influencing people's perceptions is past experience, but it is not the only one. In this research, we aim to discover how well the perceptions of the defect detection effectiveness of different techniques match their real effectiveness in the absence of prior experience. To do this, we conduct an empirical study plus a replication. During the original study, we conduct a controlled experiment with students applying two testing techniques and a code review technique. At the end of the experiment, they take a survey to find out which technique they perceive to be most effective. The results show that participants' perceptions are wrong and that this mismatch is costly in terms of quality. In order to gain further insight into the results, we replicate the controlled experiment and extend the survey to include questions about participants' opinions on the techniques and programs. The results of the replicated study confirm the findings of the original study and suggest that participants' perceptions might be based not on their opinions about complexity or preferences for techniques but on how well they think that they have applied the techniques.

Figures (10)

• Figure 1: Scatterplot for observed mismatch cost in the original study. Datapoints correspond to the mismatch cost in Table \ref{['tab:s1_cost']}
• Figure 2: Scatterplot for observed mismatch cost in the replicated study
• Figure 3: Boxplot for observed technique effectiveness in the original study
• Figure 4: Normal Q-Q plot of residuals in the original study
• Figure 5: Profile plot for program by technique interaction in the original study