Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hybrid Fault-Driven Mutation Testing for Python

Saba Alimadadi, Golnaz Gharachorlu

TL;DR

This work addresses the gap in mutation testing for Python caused by dynamic typing by introducing PyTation, a hybrid static-dynamic mutation framework with seven Python-specific operators inspired by PySStuBs anti-patterns. PyTation identifies mutation candidates through a dual analysis, prunes unlikely mutants via execution coverage, and applies targeted AST mutations to simulate runtime faults, yielding many unique and complementary mutants relative to Cosmic Ray. Empirical evaluation on 13 real-world Python projects shows PyTation uncovers test gaps even in high-coverage code, with a high average mutation score and low equivalent-mutant rates, while offering distinct fault models. The approach advances practical mutation testing for Python, enabling deeper assessment of test suites and guiding test-suite augmentation with Python-centric fault scenarios.

Abstract

Mutation testing is an effective technique for assessing the effectiveness of test suites by systematically injecting artificial faults into programs. However, existing mutation testing techniques fall short in capturing many types of common faults in dynamically typed languages like Python. In this paper, we introduce a novel set of seven mutation operators that are inspired by prevalent anti-patterns in Python programs, designed to complement the existing general-purpose operators and broaden the spectrum of simulated faults. We propose a mutation testing technique that utilizes a hybrid of static and dynamic analyses to mutate Python programs based on these operators while minimizing equivalent mutants. We implement our approach in a tool called PyTation and evaluate it on 13 open-source Python applications. Our results show that PyTation generates mutants that complement those from general-purpose tools, exhibiting distinct behaviour under test execution and uncovering inadequacies in high-coverage test suites. We further demonstrate that PyTation produces a high proportion of unique mutants, a low cross-kill rate, and a low test overlap ratio relative to baseline tools, highlighting its novel fault model. PyTation also incurs few equivalent mutants, aided by dynamic analysis heuristics.

Hybrid Fault-Driven Mutation Testing for Python

TL;DR

This work addresses the gap in mutation testing for Python caused by dynamic typing by introducing PyTation, a hybrid static-dynamic mutation framework with seven Python-specific operators inspired by PySStuBs anti-patterns. PyTation identifies mutation candidates through a dual analysis, prunes unlikely mutants via execution coverage, and applies targeted AST mutations to simulate runtime faults, yielding many unique and complementary mutants relative to Cosmic Ray. Empirical evaluation on 13 real-world Python projects shows PyTation uncovers test gaps even in high-coverage code, with a high average mutation score and low equivalent-mutant rates, while offering distinct fault models. The approach advances practical mutation testing for Python, enabling deeper assessment of test suites and guiding test-suite augmentation with Python-centric fault scenarios.

Abstract

Mutation testing is an effective technique for assessing the effectiveness of test suites by systematically injecting artificial faults into programs. However, existing mutation testing techniques fall short in capturing many types of common faults in dynamically typed languages like Python. In this paper, we introduce a novel set of seven mutation operators that are inspired by prevalent anti-patterns in Python programs, designed to complement the existing general-purpose operators and broaden the spectrum of simulated faults. We propose a mutation testing technique that utilizes a hybrid of static and dynamic analyses to mutate Python programs based on these operators while minimizing equivalent mutants. We implement our approach in a tool called PyTation and evaluate it on 13 open-source Python applications. Our results show that PyTation generates mutants that complement those from general-purpose tools, exhibiting distinct behaviour under test execution and uncovering inadequacies in high-coverage test suites. We further demonstrate that PyTation produces a high proportion of unique mutants, a low cross-kill rate, and a low test overlap ratio relative to baseline tools, highlighting its novel fault model. PyTation also incurs few equivalent mutants, aided by dynamic analysis heuristics.
Paper Structure (48 sections, 2 equations, 1 figure, 5 tables)

This paper contains 48 sections, 2 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Mutation Operators
  3. Remove Function Argument (RemFuncArg)
  4. Remove Conversion Function (RemConvFunc)
  5. Remove Element From Container (RemElCont)
  6. Remove Expression from Condition (RemExpCond)
  7. Object Oriented Mutation Operators
  8. Change Used Attribute (ChUsedAttr)
  9. Remove Attribute Access (RemAttrAcc)
  10. Remove Method Call (RemMetCall)
  11. Approach
  12. Identifying Mutation Candidates
  13. Static Analysis
  14. RemExpCond
  15. Dynamic Analysis
  16. ...and 33 more sections

Figures (1)

  • Figure 1: Overview of PyTation. The approach combines static (a) and dynamic (b) analyses to identify mutation candidates, prunes unreachable locations, and generates mutants (c) for mutation score computation (d).

Theorems & Definitions (1)

  • Definition 1: Dynamic Subsumption Graph