Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatically Removing Unnecessary Stubbings from Test Suites

Mengzhen Li, Mattia Fazzini

TL;DR

ARUS automates the removal of unnecessary stubbings in test suites by executing tests to identify unused stubbings, classifying them into totally-unnecessary, used-unnecessary-setup, and used-unnecessary-helper categories, and applying targeted removal strategies. The approach yields a practical tool implemented for Java projects using Mockito, with an empirical evaluation over 128 real-world projects showing a 98.6% resolution rate and limited increases in code complexity. ARUS's three-phase workflow—test execution, stubbing analysis, and removal—enables automated maintenance of test fixtures, and its results are appealing to developers as demonstrated by merged pull requests. The work discusses tradeoffs, limitations (notably parameterized tests and loops), and directions for future work, including alternative resolution strategies and stronger validation.

Abstract

Most modern software systems are characterized by a high number of components whose interactions can affect and complicate testing activities. During testing, developers can account for the interactions by isolating the code under test using test doubles and stubbings. During the evolution of a test suite, stubbings might become unnecessary, and developers should remove unnecessary stubbings, as their definitions can introduce unreliable test results in future versions of the test suite. Unfortunately, removing unnecessary stubbings is still a manual task that can be complex and time-consuming. To help developers in this task, we propose ARUS, a technique to automatically remove unnecessary stubbings from test suites. Given a software project and its test suite, the technique executes the tests to identify unnecessary stubbings and then removes them using different approaches based on the characteristics of the stubbings. We performed an empirical evaluation based on 128 Java projects that use Mockito for stubbing and contain 280 stubbing definitions that lead to 1,529 unnecessary stubbings. Overall, our technique provides a solution for 276 of the definitions (98.6% resolution rate), ARUS' time cost is negligible, and, on average, the technique's changes introduce a limited increase in code complexity. We submitted ARUS' changes to the projects through pull requests and 83 resolutions are already merged.

Automatically Removing Unnecessary Stubbings from Test Suites

TL;DR

ARUS automates the removal of unnecessary stubbings in test suites by executing tests to identify unused stubbings, classifying them into totally-unnecessary, used-unnecessary-setup, and used-unnecessary-helper categories, and applying targeted removal strategies. The approach yields a practical tool implemented for Java projects using Mockito, with an empirical evaluation over 128 real-world projects showing a 98.6% resolution rate and limited increases in code complexity. ARUS's three-phase workflow—test execution, stubbing analysis, and removal—enables automated maintenance of test fixtures, and its results are appealing to developers as demonstrated by merged pull requests. The work discusses tradeoffs, limitations (notably parameterized tests and loops), and directions for future work, including alternative resolution strategies and stronger validation.

Abstract

Most modern software systems are characterized by a high number of components whose interactions can affect and complicate testing activities. During testing, developers can account for the interactions by isolating the code under test using test doubles and stubbings. During the evolution of a test suite, stubbings might become unnecessary, and developers should remove unnecessary stubbings, as their definitions can introduce unreliable test results in future versions of the test suite. Unfortunately, removing unnecessary stubbings is still a manual task that can be complex and time-consuming. To help developers in this task, we propose ARUS, a technique to automatically remove unnecessary stubbings from test suites. Given a software project and its test suite, the technique executes the tests to identify unnecessary stubbings and then removes them using different approaches based on the characteristics of the stubbings. We performed an empirical evaluation based on 128 Java projects that use Mockito for stubbing and contain 280 stubbing definitions that lead to 1,529 unnecessary stubbings. Overall, our technique provides a solution for 276 of the definitions (98.6% resolution rate), ARUS' time cost is negligible, and, on average, the technique's changes introduce a limited increase in code complexity. We submitted ARUS' changes to the projects through pull requests and 83 resolutions are already merged.
Paper Structure (15 sections, 5 figures, 1 table, 2 algorithms)

This paper contains 15 sections, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Technique
  4. Test Suite Execution
  5. Stubbings Analysis
  6. Unnecessary Stubbings Removal
  7. Implementation
  8. Evaluation
  9. Benchmarks
  10. Methodology
  11. Results
  12. Discussion
  13. Limitations and Threats to Validity
  14. Related Work
  15. Conclusion

Figures (5)

  • Figure 1: Example of a test with unnecessary stubbings.
  • Figure 2: High-level overview of ARUS.
  • Figure 3: Portion of an execution info file.
  • Figure 4: Percentage variation of cognitive complexity.
  • Figure 5: Percentage variation of LOC.