SupermarQ: A Scalable Quantum Benchmark Suite

TL;DR

SupermarQ presents a principled, scalable quantum benchmark suite that combines application-level workloads with hardware-agnostic feature vectors to assess cross-platform performance. By integrating diverse benchmarks (GHZ, Mermin-Bell, ECC proxies, QAOA, VQE, Hamiltonian simulation) and a Closed Division evaluation on multiple cloud backends, it demonstrates meaningful end-to-end system characterization beyond gate- or circuit-level metrics. The work shows how workload features correlate with hardware performance, highlights the pivotal role of mid-circuit measurements and compiler mappings, and argues for an adaptable, community-driven benchmarking ecosystem to guide future quantum hardware and software co-design. Collectively, SupermarQ provides a concrete, extensible framework to benchmark current NISQ devices and near-future fault-tolerant architectures, enabling more reliable performance comparisons and progress tracking.

Abstract

The emergence of quantum computers as a new computational paradigm has been accompanied by speculation concerning the scope and timeline of their anticipated revolutionary changes. While quantum computing is still in its infancy, the variety of different architectures used to implement quantum computations make it difficult to reliably measure and compare performance. This problem motivates our introduction of SupermarQ, a scalable, hardware-agnostic quantum benchmark suite which uses application-level metrics to measure performance. SupermarQ is the first attempt to systematically apply techniques from classical benchmarking methodology to the quantum domain. We define a set of feature vectors to quantify coverage, select applications from a variety of domains to ensure the suite is representative of real workloads, and collect benchmark results from the IBM, IonQ, and AQT@LBNL platforms. Looking forward, we envision that quantum benchmarking will encompass a large cross-community effort built on open source, constantly evolving benchmark suites. We introduce SupermarQ as an important step in this direction.

Figures (4)

• Figure 1: Feature maps and sample circuits for each of the benchmarks evaluated in this study. The definitions of the Program Communication (PC), Critical Depth (CD), Entanglement-Ratio (Ent), Measurement (Mea), Parallelism (Par), and Liveness (Liv) features are given in Sec. \ref{['sec:design']}.
• Figure 2: Benchmark results evaluated across superconducting and trapped ion devices (the black X's indicate benchmarks that exceed the number of qubits available on the device). The results for each benchmark appear in the same order given along the x-axis of (g) and (h). Each bar denotes the average performance over multiple benchmark runs while the error-bars indicate a single standard deviation from the mean score. The specific score functions for each benchmark are given in Sec. \ref{['sec:benchmarks']}. In every benchmark run, we executed 2000 shots on the IBM devices, 1024 on the AQT device, and 35 on the IonQ processor. The shot counts were selected to maintain a reasonable cost budget for collecting the benchmark results.
• Figure 3: Heatmaps showing the correlation between application features and system performance. The correlations in (a) were computed using all of the benchmark data, whereas in (b) the data from the phase and bit code benchmarks was excluded.
• Figure 4: Example of the impact the error correction (EC) benchmarks have on the correlation between the application features and system performance.