Quantum Testing in the Wild: A Case Study with Qiskit Algorithms

TL;DR

The paper addresses the challenge of testing quantum software by performing an empirical analysis of the Qiskit Algorithms test suite to uncover how practitioners test quantum components. It identifies seven recurring test patterns, showing a strong reliance on classical testing techniques and limited uptake of quantum-specific testing methods; the study analyzes 309 quantum-related tests, revealing distributions such as 133 tests employing Data-Driven Testing and a mix of black-box, gray-box, and classical tests. By examining the Variational Quantum Eigensolver (VQE) workflow and circuit-level checks, it highlights concrete practices like fixed seeds, deterministic oracles, and unitary-equivalence tests for circuits, while also noting gaps in transpilation and real-hardware testing. The findings underscore a gap between academia and practice, suggesting opportunities to broaden QC testing methods across frameworks and languages (e.g., Q#, PennyLane, Cirq) to improve the reliability of quantum software in real-world deployments.

Abstract

Although classical computing has excelled in a wide range of applications, there remain problems that push the limits of its capabilities, especially in fields like cryptography, optimization, and materials science. Quantum computing introduces a new computational paradigm, based on principles of superposition and entanglement to explore solutions beyond the capabilities of classical computation. With the increasing interest in the field, there are challenges and opportunities for academics and practitioners in terms of software engineering practices, particularly in testing quantum programs. This paper presents an empirical study of testing patterns in quantum algorithms. We analyzed all the tests handling quantum aspects of the implementations in the Qiskit Algorithms library and identified seven distinct patterns that make use of (1) fixed seeds for algorithms based on random elements; (2) deterministic oracles; (3) precise and approximate assertions; (4) Data-Driven Testing (DDT); (5) functional testing; (6) testing for intermediate parts of the algorithms being tested; and (7) equivalence checking for quantum circuits. Our results show a prevalence of classical testing techniques to test the quantum-related elements of the library, while recent advances from the research community have yet to achieve wide adoption among practitioners.