Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Grained Assertion-Based Test Selection

Sijia Gu, Ali Mesbah

TL;DR

This paper introduces Selertion, the first regression test selection approach that operates at the assertion level inside test methods to increase precision and reduce test execution time. It combines static change computation, assertion-level test slicing via backward data-dependency analysis, and runtime dependency instrumentation to safely select only the assertions, tests, or classes affected by production-code changes. Across 11 open-source Java projects, Selertion achieves substantial precision gains and reduces overall test time by a median of $63\%$, outperforming state-of-the-art RTS tools, especially for longer-running test suites. Safety is maintained as mutation testing shows Selertion eliminates mutants at the same rate as the full test suite, and the approach is implemented in a Java tool that builds on Eclipse JDT with support for Maven/JUnit4 and popular assertion libraries. The work has practical implications for accelerating regression testing and improving developer understanding of test dependencies, with future extensions to broader test domains and newer JUnit versions.

Abstract

For large software applications, running the whole test suite after each code change is time- and resource-intensive. Regression test selection techniques aim at reducing test execution time by selecting only the tests that are affected by code changes. However, existing techniques select test entities at coarse granularity levels such as test class, which causes imprecise test selection and executing unaffected tests. We propose a novel approach that increases the selection precision by analyzing test code at statement level and treating test assertions as the unit for selection. We implement our fine-grained test selection approach in a tool called \toolname and evaluate it by comparing against two state-of-the-art test selection techniques using 11 open-source subjects. Our results show that \toolname increases selection precision for all the subjects. Our test selection reduces, on average, 63\% of the overall test time, making regression testing 7--38\% faster than the other techniques. Our results also indicate that subjects with longer test execution time benefit more by our fine-grained selection technique.

Fine-Grained Assertion-Based Test Selection

TL;DR

This paper introduces Selertion, the first regression test selection approach that operates at the assertion level inside test methods to increase precision and reduce test execution time. It combines static change computation, assertion-level test slicing via backward data-dependency analysis, and runtime dependency instrumentation to safely select only the assertions, tests, or classes affected by production-code changes. Across 11 open-source Java projects, Selertion achieves substantial precision gains and reduces overall test time by a median of , outperforming state-of-the-art RTS tools, especially for longer-running test suites. Safety is maintained as mutation testing shows Selertion eliminates mutants at the same rate as the full test suite, and the approach is implemented in a Java tool that builds on Eclipse JDT with support for Maven/JUnit4 and popular assertion libraries. The work has practical implications for accelerating regression testing and improving developer understanding of test dependencies, with future extensions to broader test domains and newer JUnit versions.

Abstract

For large software applications, running the whole test suite after each code change is time- and resource-intensive. Regression test selection techniques aim at reducing test execution time by selecting only the tests that are affected by code changes. However, existing techniques select test entities at coarse granularity levels such as test class, which causes imprecise test selection and executing unaffected tests. We propose a novel approach that increases the selection precision by analyzing test code at statement level and treating test assertions as the unit for selection. We implement our fine-grained test selection approach in a tool called \toolname and evaluate it by comparing against two state-of-the-art test selection techniques using 11 open-source subjects. Our results show that \toolname increases selection precision for all the subjects. Our test selection reduces, on average, 63\% of the overall test time, making regression testing 7--38\% faster than the other techniques. Our results also indicate that subjects with longer test execution time benefit more by our fine-grained selection technique.
Paper Structure (20 sections, 5 figures, 5 tables, 2 algorithms)

This paper contains 20 sections, 5 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Motivating Example
  3. Approach
  4. Change Computation
  5. Test Assertion Slicing
  6. Code Instrumentation
  7. Test Selection
  8. Implementation
  9. Evaluation
  10. Existing RTS Tools
  11. Subject Systems
  12. Procedure and Results
  13. Prevalence (RQ1).
  14. Precision (RQ2).
  15. Efficiency (RQ3).
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Test cases from Apache Commons Math. Assertion statements are highlighted.
  • Figure 2: Test cases are sliced into separate code fragments for each assertion. Fragments that depend on production method Complex.negate() are grayed-out.
  • Figure 3: Overview of our approach.
  • Figure 3: Results: testing time and comparison with other RTS tools
  • Figure 4: Distributions of selected assertion ratio. Lower values are better as fewer tests are selected.

Theorems & Definitions (1)

  • Definition 1: Assertion Slice