Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empirically Evaluating the Use of Bytecode for Diversity-Based Test Case Prioritisation

Islam T. Elgendy, Robert M. Hierons, Phil McMinn

TL;DR

This work tackles the cost of regression testing by introducing bytecode as a novel diversity artefact for Test Case Prioritisation (TCP). It adapts two static TCP approaches (Ledru-TCP and FAST-TCP) to operate on test-case bytecode, with a filtering step to remove routine instructions, and compares against traditional textual diversity on seven Defects4J projects. The findings show that bytecode-based TCP yields higher or comparable fault-detection performance (APFD and real-fault detection) while delivering massive speedups in preparation and prioritisation, especially with filtering. Although dynamic, coverage-based TCP approaches still achieve better fault detection, the bytecode-based static approaches offer a practical, scalable alternative in resource-constrained environments and when coverage instrumentation is unavailable.

Abstract

Regression testing assures software correctness after changes but is resource-intensive. Test Case Prioritisation (TCP) mitigates this by ordering tests to maximise early fault detection. Diversity-based TCP prioritises dissimilar tests, assuming they exercise different system parts and uncover more faults. Traditional static diversity-based TCP approaches (i.e., methods that utilise the dissimilarity of tests), like the state-of-the-art FAST approach, rely on textual diversity from test source code, which is effective but inefficient due to its relative verbosity and redundancies affecting similarity calculations. This paper is the first to study bytecode as the basis of diversity in TCP, leveraging its compactness for improved efficiency and accuracy. An empirical study on seven Defects4J projects shows that bytecode diversity improves fault detection by 2.3-7.8% over text-based TCP. It is also 2-3 orders of magnitude faster in one TCP approach and 2.5-6 times faster in FAST-based TCP. Filtering specific bytecode instructions improves efficiency up to fourfold while maintaining effectiveness, making bytecode diversity a superior static approach.

Empirically Evaluating the Use of Bytecode for Diversity-Based Test Case Prioritisation

TL;DR

This work tackles the cost of regression testing by introducing bytecode as a novel diversity artefact for Test Case Prioritisation (TCP). It adapts two static TCP approaches (Ledru-TCP and FAST-TCP) to operate on test-case bytecode, with a filtering step to remove routine instructions, and compares against traditional textual diversity on seven Defects4J projects. The findings show that bytecode-based TCP yields higher or comparable fault-detection performance (APFD and real-fault detection) while delivering massive speedups in preparation and prioritisation, especially with filtering. Although dynamic, coverage-based TCP approaches still achieve better fault detection, the bytecode-based static approaches offer a practical, scalable alternative in resource-constrained environments and when coverage instrumentation is unavailable.

Abstract

Regression testing assures software correctness after changes but is resource-intensive. Test Case Prioritisation (TCP) mitigates this by ordering tests to maximise early fault detection. Diversity-based TCP prioritises dissimilar tests, assuming they exercise different system parts and uncover more faults. Traditional static diversity-based TCP approaches (i.e., methods that utilise the dissimilarity of tests), like the state-of-the-art FAST approach, rely on textual diversity from test source code, which is effective but inefficient due to its relative verbosity and redundancies affecting similarity calculations. This paper is the first to study bytecode as the basis of diversity in TCP, leveraging its compactness for improved efficiency and accuracy. An empirical study on seven Defects4J projects shows that bytecode diversity improves fault detection by 2.3-7.8% over text-based TCP. It is also 2-3 orders of magnitude faster in one TCP approach and 2.5-6 times faster in FAST-based TCP. Filtering specific bytecode instructions improves efficiency up to fourfold while maintaining effectiveness, making bytecode diversity a superior static approach.

Paper Structure

This paper contains 24 sections, 1 equation, 4 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Test Case Prioritisation
  4. Diversity-Based TCP
  5. Bytecode
  6. Approach
  7. Motivating Example
  8. Filtering Bytecode
  9. TCP Approaches
  10. Evaluation
  11. Research Questions
  12. Evaluation Metrics
  13. Subjects
  14. Methodology
  15. Data Collection
  16. ...and 9 more sections

Figures (4)

  • Figure 1: TCP process diagram
  • Figure 2: An example of four test cases from the Math project, with test case 3 having two variations
  • Figure 3: The human-readable and the Hexadecimal formats of the bytecodes for the tests in Figure \ref{['fig:example']}
  • Figure 4: The preparation times in seconds for all static TCP approaches