Who cares about testing?: Co-creations of Socio-technical Software Testing Experiences

TL;DR

The paper investigates why software developers do or do not engage in testing by eliciting lived experiences through socio-technical grounded theory (STGT) and 19 in-depth interviews. It demonstrates that testing decisions emerge from a recursive interplay of technical, organizational, and social conditions, rather than from linear rationales. The authors introduce three concepts—TestingSignatures, TestingEchoes, and TestingEfficacy—and formulate the theory of Emerging Testing Strategies (ETS), linking artefacts, conversations, and perceived value to testing adoption and adaptation. The work reframes testing as a socio-technical experience where collective reflection and material infrastructures co-create testing cultures and practices, with implications for practice and future research in software engineering.

Abstract

Software testing is crucial for ensuring software quality, yet developers' engagement with it varies widely. Identifying the technical, organizational and social factors that lead to differences in engagement is required to remove barriers and utilize enablers for testing. While much research emphasizes the usefulness of testing strategies and technical solutions, less is known about why developers do (not) test. This study investigates the lived experience of software developers to illuminate how their opinions about testing change. Learning about personal evolutions of practice, we explore when and why testing is used. Employing socio-technical grounded theory (STGT), we construct a theory by systematically analyzing data from 19 in-depth, semi-structured interviews with software developers. Allowing interviewees to reflect on how and why they approach software testing, we explore perspectives that are rooted in their contextual experiences. We develop eleven categories of circumstances that act as conditions for the application and adaptation of testing practices and introduce three concepts that we then use to present our theory of emerging testing strategies (ETS) that explains why developers do (not) use testing practices. This study reveals a new perspective on the connection between testing artifacts and collective reflection of practitioners, and it embraces testing as an experience in which human- and social aspects are entangled with organizational and technical circumstances.

Figures (8)

• Figure 1: Illustration of the first phases of our STGT research design hoda_qualitative_2024. We structure our research design in four phases. The Study Preparation- and Basic data collection and analysis (DCA) phases are illustrated here. The consecutive phases of Advanced DCA and Theoretical Structuring and Reporting are illustrated in \ref{['fig:advanced_stage']}. Circled numbers to are referenced in the text.
• Figure 2: Illustration of the last two phases of our research design. We reach the advanced stage when data collection and analysis in the basic stage (refer to \ref{['fig:basic_stage']}) allows us us to establish links between preliminary categories . In the advanced stage for data collection and analysis we focus our work and begin the process of systematic theory construction.
• Figure 3: Diagram illustrating the mutual, circular influence of #TestingSignatures and #TestingEchoes. For example, a conversation between two developers (#testingEcho) about testing leads to experimentation and eventually to the creation of a new unit test (#testingSignature). In turn, working with a unit test causes a developer to reflect. Those reflections about how and why the unit test does (not) fit into the project's testing strategy leads to new conversations in code reviews, meetings or informal settings. #testingEfficacy influences this mutual influence. The presence of #testingSignatures impacts the efficiency with which testing contributions can be made; existing unit tests can be copied. Conversations explicate the value of testing. In turn, the empowerment of developers to contribute to testing efforts in a way that feels efficient and appreciated (#testingEfficacy). encourages developers to reflect and gives them the ability to experiment.
• Figure 4: Depiction of the testing process as a black-box with recursive dependency $y=T(x,z)$. #TestingEfficacy ($z$) influences the outcome of the testing decision making process $y$, and the potential of #TestingEchoes and #testingSignatures on efficacy for testing in the future ($z'$). As the actor has no full insight on $z$, their action cascade recursively and largely unpredictable von_foerster_understanding_2003
• Figure 5: Illustration of an example of the recursive evolution of testing strategies. A developer implementing a feature unavoidably affects a projects testing strategy and future attempts at software testing. Each action $x$, like the implementation of a new feature leads to a testing related output $y$. The developers' decision to implement tests is affected by organizational, social and technical affordances $z$. Apart from the outcome $y$, the actions of the developer alter the conditions for testing ($Z\rightarrow z'$). For example, the developer presents the benefits of their solution for testing in a meeting which impacts how peers perceive testing (change in testing culture). The implementation of the next feature $x'$ is influenced by those altered factors ($y'=T(x',z')$ with $z'=Z(x,z))$.