Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Test Behaviors, Not Methods! Detecting Tests Obsessed by Methods

Andre Hora, Andy Zaidman

TL;DR

This paper introduces Test Obsessed by Method, a runtime-based test smell for detecting tests that exercise multiple execution paths of a single production method, arguing that such tests likely verify multiple behaviors. It positions this smell as a complement to the Eager Test concept, which relies on static method-call counts, and demonstrates that path coverage provides a more semantically meaningful signal. Through an initial study of 2,054 test methods from 12 Python Standard Library suites, the authors identify 44 true-positive smelly tests (81.5% precision) across 11 suites, and show these could be refactored into 118 more focused tests, often with comments acknowledging distinct behaviors. The work highlights the practicality of runtime analysis (via SpotFlow) for smell detection, discusses how to fix detected tests, and outlines limitations and directions for future research to validate and generalize the approach.

Abstract

Best testing practices state that tests should verify a single functionality or behavior of the system. Tests that verify multiple behaviors are harder to understand, lack focus, and are more coupled to the production code. An attempt to identify this issue is the test smell \emph{Eager Test}, which aims to capture tests that verify too much functionality based on the number of production method calls. Unfortunately, prior research suggests that counting production method calls is an inaccurate measure, as these calls do not reliably serve as a proxy for functionality. We envision a complementary solution based on runtime analysis: we hypothesize that some tests that verify multiple behaviors will likely cover multiple paths of the same production methods. Thus, we propose a novel test smell named \emph{Test Obsessed by Method}, a test method that covers multiple paths of a single production method. We provide an initial empirical study to explore the presence of this smell in 2,054 tests provided by 12 test suites of the Python Standard Library. (1) We detect 44 \emph{Tests Obsessed by Methods} in 11 of the 12 test suites. (2) Each smelly test verifies a median of two behaviors of the production method. (3) The 44 smelly tests could be split into 118 novel tests. (4) 23% of the smelly tests have code comments recognizing that distinct behaviors are being tested. We conclude by discussing benefits, limitations, and further research.

Test Behaviors, Not Methods! Detecting Tests Obsessed by Methods

TL;DR

This paper introduces Test Obsessed by Method, a runtime-based test smell for detecting tests that exercise multiple execution paths of a single production method, arguing that such tests likely verify multiple behaviors. It positions this smell as a complement to the Eager Test concept, which relies on static method-call counts, and demonstrates that path coverage provides a more semantically meaningful signal. Through an initial study of 2,054 test methods from 12 Python Standard Library suites, the authors identify 44 true-positive smelly tests (81.5% precision) across 11 suites, and show these could be refactored into 118 more focused tests, often with comments acknowledging distinct behaviors. The work highlights the practicality of runtime analysis (via SpotFlow) for smell detection, discusses how to fix detected tests, and outlines limitations and directions for future research to validate and generalize the approach.

Abstract

Best testing practices state that tests should verify a single functionality or behavior of the system. Tests that verify multiple behaviors are harder to understand, lack focus, and are more coupled to the production code. An attempt to identify this issue is the test smell \emph{Eager Test}, which aims to capture tests that verify too much functionality based on the number of production method calls. Unfortunately, prior research suggests that counting production method calls is an inaccurate measure, as these calls do not reliably serve as a proxy for functionality. We envision a complementary solution based on runtime analysis: we hypothesize that some tests that verify multiple behaviors will likely cover multiple paths of the same production methods. Thus, we propose a novel test smell named \emph{Test Obsessed by Method}, a test method that covers multiple paths of a single production method. We provide an initial empirical study to explore the presence of this smell in 2,054 tests provided by 12 test suites of the Python Standard Library. (1) We detect 44 \emph{Tests Obsessed by Methods} in 11 of the 12 test suites. (2) Each smelly test verifies a median of two behaviors of the production method. (3) The 44 smelly tests could be split into 118 novel tests. (4) 23% of the smelly tests have code comments recognizing that distinct behaviors are being tested. We conclude by discussing benefits, limitations, and further research.
Paper Structure (14 sections, 4 figures, 2 tables)

This paper contains 14 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tests Obsessed by Methods
  4. Overview
  5. Detecting Tests Obsessed by Methods
  6. Preliminary Empirical Study
  7. Design
  8. Results
  9. Discussion
  10. Fixing Tests Obsessed by Methods
  11. Runtime Analysis
  12. Comparison with Eager Test
  13. Limitations
  14. Conclusion and Future Work

Figures (4)

  • Figure 1: Example of a Test Obsessed by Method in CPython. Test test_setfirstweekday covers three paths of setfirstweekday. This test could be split into three tests.
  • Figure 2: Test Obsessed by Method: test_constructor tests two behaviors of Namespace (argparse).
  • Figure 3: Test Obsessed by Method: test_splitroot tests three behaviors of splitroot (pathlib).
  • Figure 4: Test Obsessed by Method: test_parsing_error tests three behaviors of ParsingError (configparser).