Application-Oriented Performance Benchmarks for Quantum Computing

TL;DR

This paper presents an extensible, open-source suite of application-oriented benchmarks to evaluate quantum hardware performance on realistic tasks, using volumetric benchmarking to visualize fidelity as circuit width and depth vary. It builds on component-level metrics and the quantum volume, but emphasizes end-to-end application performance across multiple algorithms, languages, and hardware platforms. The results demonstrate that volumetric extrapolations from quantum volume can predict some hardware performance, yet device connectivity, compiler choices, and error models can cause deviations, underscoring the need for diverse, evolving benchmarks. By measuring both result fidelity and quantum execution time, the framework provides practical markers for progress toward useful quantum advantage and helps users compare devices and stacks in a realistic, end-to-end manner.

Abstract

In this work we introduce an open source suite of quantum application-oriented performance benchmarks that is designed to measure the effectiveness of quantum computing hardware at executing quantum applications. These benchmarks probe a quantum computer's performance on various algorithms and small applications as the problem size is varied, by mapping out the fidelity of the results as a function of circuit width and depth using the framework of volumetric benchmarking. In addition to estimating the fidelity of results generated by quantum execution, the suite is designed to benchmark certain aspects of the execution pipeline in order to provide end-users with a practical measure of both the quality of and the time to solution. Our methodology is constructed to anticipate advances in quantum computing hardware that are likely to emerge in the next five years. This benchmarking suite is designed to be readily accessible to a broad audience of users and provides benchmarks that correspond to many well-known quantum computing algorithms.

Figures (37)

• Figure 1: Quantum Application-Oriented Performance Benchmarks. The results of executing our quantum application-oriented performance benchmarking suite on a simulator of a noisy quantum computer, with results split into benchmarks based on three loose categories of algorithm: tutorial, subroutine, and functional. For each benchmark, circuits are run for a variety of problem sizes. This typically correspond to the circuit's width, i.e., the number of qubits it acts on, which here range from 2 to 12 qubits. The result fidelity, a measure of the result quality, is computed for each circuit execution, and is shown as a colored square positioned at the corresponding circuit's width and normalized depth. Results for circuits of equal width and similar depth are averaged together. The results of the application-oriented benchmarks are shown on top of a 'volumetric background' (grey-scale squares). Here and throughout this paper, except where stated, this volumetric background is a heuristic extrapolation of a device's quantum volume (here, 32) to predict the region in which a circuit's result fidelity will be above $1/2$ (the grey squares). Note that this extrapolation is not expected to always be accurate (see discussion in main text) but it is nevertheless useful. This is because any deviations between the performance of the algorithmic benchmarks and the prediction of the volumetric background signify that, for the processor in question, the performance of these algorithms is difficult to predict from the processor's quantum volume alone.
• Figure 2: An example of component-level performance metrics. The component-level performance metrics provided by Amazon Braket for the IonQ Quantum Processing Unit.
• Figure 3: An example of component-level performance metrics. The component-level performance metrics provided by Google for its Weber device.
• Figure 4: Quantum volume examples. The number of qubits and the quantum volume for a selection of IBM Q machines, accessed via IBM Quantum Services.
• Figure 5: An example of volumetric benchmarking. Volumetric benchmarking results for a hypothetical 12-qubit quantum computer, whereby a quantum computer's performance on some family of circuits is mapped out as a function of circuit width and depth. This example uses a binary measure of performance: grey and white squares show the circuit shapes at which the test circuits succeeded and failed, respectively, according to this metric. The location of the quantum volume circuits, that would be used to extract the quantum volume, are shown in bold. Here and throughout, 'depth' for all circuits is defined with respect to a particular standardized gate set, and in this gate set the quantum volume circuits are not square.