Understanding and Characterizing Mock Assertions in Unit Tests

TL;DR

This paper presents the first empirical study of mock assertions in unit tests, examining how developers use mock assertions with test doubles across 11 popular Java projects. It shows that while 41% of test cases involving test doubles employ mock assertions, only about 9% of mock invocations are actually verified, indicating selective and purpose-driven use. The study identifies three key method categories commonly verified by mock assertions—external resources, state mutators, and callbacks—and two interaction types: control-flow and data-flow. It further demonstrates that mock assertions complement traditional test assertions in fault detection, with many mutants detectable only by mock verifications, and offers practical guidance for integrating mock assertions into test generation and maintenance while avoiding overly aggressive mocking.

Abstract

Mock assertions provide developers with a powerful means to validate program behaviors that are unobservable to test assertions. Despite their significance, they are rarely considered by automated test generation techniques. Effective generation of mock assertions requires understanding how they are used in practice. Although previous studies highlighted the importance of mock assertions, none provide insight into their usages. To bridge this gap, we conducted the first empirical study on mock assertions, examining their adoption, the characteristics of the verified method invocations, and their effectiveness in fault detection. Our analysis of 4,652 test cases from 11 popular Java projects reveals that mock assertions are mostly applied to validating specific kinds of method calls, such as those interacting with external resources and those reflecting whether a certain code path was traversed in systems under test. Additionally, we find that mock assertions complement traditional test assertions by ensuring the desired side effects have been produced, validating control flow logic, and checking internal computation results. Our findings contribute to a better understanding of mock assertion usages and provide a foundation for future related research such as automated test generation that support mock assertions.

Figures (8)

• Figure 1: An Illustration of Mock Assertions and Test Assertions in Unit Testing. Mock assertions are leveraged to check if the URL used for downloading is correct and whether the resource has been saved to the database.
• Figure 2: A Mock Assertion Verifying an Invocation to a Method that Writes Data to a Netty Channel (Project Dubbo). Developers check the argument to ensure the correctness of the data being written.
• Figure 3: A Mock Assertion Verifying an Invocation to a State Mutator that Inserts a Header to an HTTP Response (Project Neo4j). Developers check the correctness of the arguments.
• Figure 4: A Mock Assertion Verifying an Invocation of a Callback Method Triggered During Library Processing (Project SpringBoot). Developers further check if the library passed to the callback is correct.
• Figure 5: A Mock Assertion Verifying a Method Call that is Made Conditionally (Project Camel). The method invocation is guarded by multiple branch conditions and loops.