Quantum Circuit Mutants: Empirical Analysis and Recommendations

TL;DR

The paper tackles the challenge of evaluating quantum software testing by conducting a large-scale empirical study of over 700K quantum circuit mutants generated from 382 real circuits. Using Muskit/QMutPy-based mutation, it analyzes how mutation operators, circuit attributes, and algorithm types influence fault survivability, and introduces a dataset and a recommendation tool to help researchers and practitioners create mutants with controllable difficulty. Key findings show Add mutations and faults at circuit ends tend to survive longer, while dominant-output algorithms generally exhibit higher survivability; circuit complexity shows limited predictive value for survivability. The work provides practical benchmarks and a scalable framework for systematic quantum mutation analysis, with implications for designing cost-effective testing strategies and guiding future research toward real-hardware validation and equivalence reduction methods.

Abstract

As a new research area, quantum software testing lacks systematic testing benchmarks to assess testing techniques' effectiveness. Recently, some open-source benchmarks and mutation analysis tools have emerged. However, there is insufficient evidence on how various quantum circuit characteristics (e.g., circuit depth, number of quantum gates), algorithms (e.g., Quantum Approximate Optimization Algorithm), and mutation characteristics (e.g., mutation operators) affect the detection of mutants in quantum circuits. Studying such relations is important to systematically design faulty benchmarks with varied attributes (e.g., the difficulty in detecting a seeded fault) to facilitate assessing the cost-effectiveness of quantum software testing techniques efficiently. To this end, we present a large-scale empirical evaluation with more than 700K faulty benchmarks (quantum circuits) generated by mutating 382 real-world quantum circuits. Based on the results, we provide valuable insights for researchers to define systematic quantum mutation analysis techniques. We also provide a tool to recommend mutants to users based on chosen characteristics (e.g., a quantum algorithm type) and the required difficulty of detecting mutants. Finally, we also provide faulty benchmarks that can already be used to assess the cost-effectiveness of quantum software testing techniques.

Figures (8)

• Figure 1: A quantum circuit example. The circuit has five qubits (i.e., $q[0]$ to $q[4]$) and a group of five classical bits (i.e., together denoted as $c_{5}$). The measurements collapse the state of all qubits to classical bits. The three selected operators are: Add a gate before position 8 in the circuit, Remove the Hadamard gate at position 0, and Replace the NOT gate with a Hadamard gate at position 13.
• Figure 2: Descriptive statistics of the generated benchmarks
• Figure 3: Average SR of all faulty benchmarks in terms of each mutation characteristic -- RQ1.1
• Figure 4: Interaction effects between Position and all other mutation characteristics -- RQ1.2. Each cell shows the SR corresponding to a specific interaction, with a darker (or lighter) blue indicating a higher (or lower) SR.
• Figure 5: Interaction effects between Operator and all other mutation characteristics -- RQ1.2. Each cell shows the SR corresponding to a specific interaction. A darker (or lighter) blue indicates a higher (or lower) SR; a white empty cell denotes an absolute zero SR; a cell with zero in it denotes a very-near-zero positive number; a cell with X tells that no benchmarks can be generated with the given combination.