Quality, Speed, and Scale: three key attributes to measure the performance of near-term quantum computers

TL;DR

This paper argues that quantum computer performance should be assessed with holistic benchmarks that reflect real-world quantum-classical workloads. It defines three core metrics—qubit count (scale), quantum volume (quality), and CLOPS (speed)—and details how offline and runtime compilation, along with classical processing, influence these measures. The authors apply CLOPS to IBM devices to reveal bottlenecks in runtime compilation and data transfer, and provide depth-1 analysis as a scalable proxy for speed. Overall, the work emphasizes integrated benchmarking to drive hardware and software improvements toward practical quantum advantage.

Abstract

Defining the right metrics to properly represent the performance of a quantum computer is critical to both users and developers of a computing system. In this white paper, we identify three key attributes for quantum computing performance: quality, speed, and scale. Quality and scale are measured by quantum volume and number of qubits, respectively. We propose a speed benchmark, using an update to the quantum volume experiments that allows the measurement of Circuit Layer Operations Per Second (CLOPS) and identify how both classical and quantum components play a role in improving performance. We prescribe a procedure for measuring CLOPS and use it to characterize the performance of some IBM Quantum systems.

Figures (6)

• Figure 1: Runtime architecture and phases of compilation. The circuit pattern of a Quantum Volume benchmark is shown, as well as its offline compilation. Circuit parameters in the Circuit Layer Operations per Second benchmark are updated during runtime, making the metric heavily dependent on the runtime architecture and runtime compilation.
• Figure 2: Benchmarking pyramid showing how quality and speed can be benchmarked. Higher-level benchmarks capture more complexity but less specificity. There may be tradeoffs between the two faces of the pyramid.
• Figure 3: Matrix of circuits used for CLOPS benchmark There are $M=100$ independent templates of QV circuits, with $D$ layers of SU(4)s, where each SU(4) in the circuit is fully parameterized. The parameters for each circuit are updated $K=10$ times. The parameters $\theta_{m,k}$ depend on the output from circuit using parameters $\theta_{m,k-1}$
• Figure 4: CLOPS results
• Figure 5: CLOPS time breakdown All times are in seconds