Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Methodological Analysis of Empirical Studies in Quantum Software Testing

Yuechen Li, Minqi Shao, Jianjun Zhao, Qichen Wang

TL;DR

This paper tackles the lack of methodological consensus in empirical quantum software testing (QST) by conducting a systematic review of 59 primary QST studies (2015–2025) from a pool of 384. It defines ten research questions spanning program under test, testing setups, evaluation metrics, and experimental resources, and provides a structured, reproducible extraction framework. Key findings reveal broad diversity in PUT types, prevalent mutation-based fault generation, uneven scalability coverage, varied test-input and oracle designs, and frequent reliance on ideal simulators, with growing but uneven open-source artifacts and benchmarks. The work offers pragmatic recommendations for aligning test design with real-world requirements, explicit treatment of test oracles, scalable reporting, and transparent artifact sharing to bolster reproducibility and cumulative progress in QST.

Abstract

In quantum software engineering (QSE), quantum software testing (QST) has attracted increasing attention as quantum software systems grow in scale and complexity. Since QST evaluates quantum programs through execution under designed test inputs, empirical studies are widely used to assess the effectiveness of testing approaches. However, the design and reporting of empirical studies in QST remain highly diverse, and a shared methodological understanding has yet to emerge, making it difficult to interpret results and compare findings across studies. This paper presents a methodological analysis of empirical studies in QST through a systematic examination of 59 primary studies identified from a literature pool of size 384. We organize our analysis around ten research questions that cover key methodological dimensions of QST empirical studies, including objects under test, baseline comparison, testing setup, experimental configuration, and tool and artifact support. Through cross-study analysis along these dimensions, we characterize current empirical practices in QST, identify recurring limitations and inconsistencies, and highlight open methodological challenges. Based on our findings, we derive insights and recommendations to inform the design, execution, and reporting of future empirical studies in QST.

A Methodological Analysis of Empirical Studies in Quantum Software Testing

TL;DR

This paper tackles the lack of methodological consensus in empirical quantum software testing (QST) by conducting a systematic review of 59 primary QST studies (2015–2025) from a pool of 384. It defines ten research questions spanning program under test, testing setups, evaluation metrics, and experimental resources, and provides a structured, reproducible extraction framework. Key findings reveal broad diversity in PUT types, prevalent mutation-based fault generation, uneven scalability coverage, varied test-input and oracle designs, and frequent reliance on ideal simulators, with growing but uneven open-source artifacts and benchmarks. The work offers pragmatic recommendations for aligning test design with real-world requirements, explicit treatment of test oracles, scalable reporting, and transparent artifact sharing to bolster reproducibility and cumulative progress in QST.

Abstract

In quantum software engineering (QSE), quantum software testing (QST) has attracted increasing attention as quantum software systems grow in scale and complexity. Since QST evaluates quantum programs through execution under designed test inputs, empirical studies are widely used to assess the effectiveness of testing approaches. However, the design and reporting of empirical studies in QST remain highly diverse, and a shared methodological understanding has yet to emerge, making it difficult to interpret results and compare findings across studies. This paper presents a methodological analysis of empirical studies in QST through a systematic examination of 59 primary studies identified from a literature pool of size 384. We organize our analysis around ten research questions that cover key methodological dimensions of QST empirical studies, including objects under test, baseline comparison, testing setup, experimental configuration, and tool and artifact support. Through cross-study analysis along these dimensions, we characterize current empirical practices in QST, identify recurring limitations and inconsistencies, and highlight open methodological challenges. Based on our findings, we derive insights and recommendations to inform the design, execution, and reporting of future empirical studies in QST.
Paper Structure (81 sections, 1 equation, 18 figures, 13 tables)

This paper contains 81 sections, 1 equation, 18 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Basics of Quantum Information Theory
  4. Quantum Programs and Their Implementation
  5. Research Methodology
  6. Research Questions
  7. Paper Collection
  8. Inclusion Criteria
  9. Keyword Search
  10. Snowballing Search
  11. Data Extraction
  12. Bibliometric Analysis
  13. Temporal Trend
  14. Publication Venues
  15. Fine-grained SE Problems
  16. ...and 66 more sections

Figures (18)

  • Figure 1: Bibliometric analysis of the 59 primary studies
  • Figure 2: Example of PUTs and the corresponding CUT for a 3-qubit Quantum Fourier Transform
  • Figure 3: Quantum algorithms and subroutines used in the primary studies
  • Figure 4: Boxplots for the numbers of adopted quantum programs
  • Figure 5: Statistics related to the buggy variants for fault detection
  • ...and 13 more figures

Theorems & Definitions (2)

  • Definition 1: Test input
  • Definition 2: Test oracle