MOTIF: A tool for Mutation Testing with Fuzzing
Jaekwon Lee, Enrico Viganò, Fabrizio Pastore, Lionel Briand
TL;DR
The paper addresses the challenge of mutation testing for embedded CPS software written in C, where symbolic execution often fails to generate effective inputs. It introduces MOTIF, a workflow that employs grey-box fuzzing with AFL++ to automatically generate unit-test drivers that compare outputs of original and mutant functions, enabling differential mutation detection. Empirical results show MOTIF outperforms a state-of-the-art KLEE-based approach, with substantial gains in mutant kill rates across several libraries and scenarios where symbolic execution is infeasible. The work demonstrates the practical viability of fuzzing-driven mutation testing for safety-critical CPS software and outlines future work toward extending the approach to C++ and exploring configuration optimizations, supported by replication resources.
Abstract
Mutation testing consists of generating test cases that detect faults injected into software (generating mutants) which its original test suite could not. By running such an augmented set of test cases, it may discover actual faults that may have gone unnoticed with the original test suite. It is thus a desired practice for embedded software running in safety-critical cyber-physical systems (CPS). Unfortunately, the state-of-the-art tool targeting C, a typical language for CPS software, relies on symbolic execution, whose limitations often prevent its application. MOTIF overcomes such limitations by leveraging grey-box fuzzing tools to generate unit test cases in C that detect injected faults in mutants. Indeed, fuzzing tools automatically generate inputs by exercising the compiled version of the software under test guided by coverage feedback, thus overcoming the limitations of symbolic execution. Our empirical assessment has shown that it detects more faults than symbolic execution (i.e., up to 47 percentage points), when the latter is applicable.