Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring a Test Data-Driven Method for Selecting and Constraining Metamorphic Relations

Alejandra Duque-Torres, Dietmar Pfahl, Claus Klammer, Stefan Fischer

TL;DR

This work tackles the challenge of selecting and constraining metamorphic relations (MRs) for metamorphic testing without relying on labeled applicability data. It introduces MetaTrimmer, a three-step, test data-driven workflow that uses fuzz-generated inputs, MT processing with MR transformations, and manual constraint derivation to identify when MRs apply and to delineate input spaces where they hold. Compared to PMR, MetaTrimmer avoids labeled datasets and accounts for data-dependent MR applicability, showing promising MR selection results (including 100% compliance for GT=1 cases) and the ability to generate actionable MR constraints from mixed cases. The approach demonstrates domain-agnostic potential and integration with fuzzing, offering practical benefits for MR coverage and test suite optimization, while acknowledging the need for automation and broader validation across domains.

Abstract

Identifying and selecting high-quality Metamorphic Relations (MRs) is a challenge in Metamorphic Testing (MT). While some techniques for automatically selecting MRs have been proposed, they are either domain-specific or rely on strict assumptions about the applicability of a pre-defined MRs. This paper presents a preliminary evaluation of MetaTrimmer, a method for selecting and constraining MRs based on test data. MetaTrimmer comprises three steps: generating random test data inputs for the SUT (Step 1), performing test data transformations and logging MR violations (Step 2), and conducting manual inspections to derive constraints (Step 3). The novelty of MetaTrimmer is its avoidance of complex prediction models that require labeled datasets regarding the applicability of MRs. Moreover, MetaTrimmer facilitates the seamless integration of MT with advanced fuzzing for test data generation. In a preliminary evaluation, MetaTrimmer shows the potential to overcome existing limitations and enhance MR effectiveness.

Exploring a Test Data-Driven Method for Selecting and Constraining Metamorphic Relations

TL;DR

This work tackles the challenge of selecting and constraining metamorphic relations (MRs) for metamorphic testing without relying on labeled applicability data. It introduces MetaTrimmer, a three-step, test data-driven workflow that uses fuzz-generated inputs, MT processing with MR transformations, and manual constraint derivation to identify when MRs apply and to delineate input spaces where they hold. Compared to PMR, MetaTrimmer avoids labeled datasets and accounts for data-dependent MR applicability, showing promising MR selection results (including 100% compliance for GT=1 cases) and the ability to generate actionable MR constraints from mixed cases. The approach demonstrates domain-agnostic potential and integration with fuzzing, offering practical benefits for MR coverage and test suite optimization, while acknowledging the need for automation and broader validation across domains.

Abstract

Identifying and selecting high-quality Metamorphic Relations (MRs) is a challenge in Metamorphic Testing (MT). While some techniques for automatically selecting MRs have been proposed, they are either domain-specific or rely on strict assumptions about the applicability of a pre-defined MRs. This paper presents a preliminary evaluation of MetaTrimmer, a method for selecting and constraining MRs based on test data. MetaTrimmer comprises three steps: generating random test data inputs for the SUT (Step 1), performing test data transformations and logging MR violations (Step 2), and conducting manual inspections to derive constraints (Step 3). The novelty of MetaTrimmer is its avoidance of complex prediction models that require labeled datasets regarding the applicability of MRs. Moreover, MetaTrimmer facilitates the seamless integration of MT with advanced fuzzing for test data generation. In a preliminary evaluation, MetaTrimmer shows the potential to overcome existing limitations and enhance MR effectiveness.
Paper Structure (15 sections, 3 figures, 4 tables)

This paper contains 15 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Metamorphic Testing
  4. Test Data Generation
  5. Predicting Metamorphic Relations
  6. Methodology
  7. MetaTrimmer
  8. System under test and pre-defined set of MRs
  9. Results and Discussion
  10. RQ$_{1}$: How well does the proposed method perform compared to the PMR approach in terms of MR selection?
  11. RQ$_{2}$: How can test data be used to obtain constraints for MRs?
  12. Threats to Validity
  13. Remarks on General Relevance
  14. Related work
  15. Conclusions

Figures (3)

  • Figure 1: MetaTrimmer overview workflow
  • Figure 2: Possible implementation pipeline of MetaTrimmer.
  • Figure 3: GT values for MetaTrimmer evaluation: 1 means the MR always applies, while 0 means it doesn't always apply. We use symbols ✓ and ✗ to indicate the percentage of runs where the MR applies or is violated. ✓ at 100% means the GT is correct. Otherwise, it's incorrect. If ✓ is 0, GT is fully incorrect; if it's less than 100% but not 0, it's partially incorrect (mixed case). Similarly, ✗ at 100% means the GT is correct. Otherwise, it could be partially incorrect (mixed case), but if ✗ is 0, the GT is incorrect.