On the Rationale and Use of Assertion Messages in Test Code: Insights from Software Practitioners

TL;DR

The paper tackles how practitioners rationalize, construct, maintain, and use assertion messages in unit tests. It employs a human-subject survey (n=138 complete responses from 217 led) to reveal that most practitioners value assertion messages for troubleshooting, understandability, and documentation, yet usage is not universal. Key contributions include empirically grounded guidelines for crafting high-quality messages, insights into which code elements shape messages, and evidence that maintenance and integration with debugging workflows are essential considerations for testing tooling. The findings motivate improvements in testing frameworks and IDEs to support real-time feedback on message quality and to better embed assertion messages within debugging processes, with implications for education and future automated test-generation research.

Abstract

Unit testing is an important practice that helps ensure the quality of a software system by validating its behavior through a series of test cases. Core to these test cases are assertion statements, which enable software practitioners to validate the correctness of the system's behavior. To aid with understanding and troubleshooting test case failures, practitioners can include a message (i.e., assertion message) within the assertion statement. While prior studies have examined the frequency and structure of assertion messages by mining software repositories, they do not determine their types or purposes or how practitioners perceive the need for or the usage of various types of assertion messages. In this paper, we survey 138 professional software practitioners to gather insights into their experience and views regarding assertion messages. Our findings reveal that a majority of survey respondents find assertion messages valuable for troubleshooting failures, improving test understandability, and serving as documentation. However, not all respondents consistently include messages in their assertion methods. We also identified common considerations for constructing effective assertion messages, challenges in crafting them, maintenance techniques, and their integration into debugging processes. Our results contribute to the understanding of current practices and provide guidelines for authoring high-quality assertion messages, serving as a foundation for best practices and coding standards. Furthermore, the insights can guide the improvement of automated unit testing tools by incorporating checks for the presence and quality of assertion messages and providing real-time feedback to practitioners.

Figures (1)

• Figure 1: Frequency of including messages in assertion methods.