Benchmarking Quantum Computers via Protocols, Comparing IBM's Heron vs IBM's Eagle

TL;DR

A protocol-based benchmarking methodology that utilizes well-defined quantumness thresholds provides a transparent and intuitive assessment of whether specific quantum processors, or isolated sub-chips within them, can demonstrate a practical quantum advantage.

Abstract

As quantum computing hardware rapidly advances, objectively evaluating the capabilities and error rates of new processors remains a critical challenge for the field. A clear and realistic understanding of current quantum performance is essential to guide research priorities and drive meaningful progress. In this work, we apply and extend a protocol-based benchmarking methodology (presented in arXiv:2505.12441) that utilizes well-defined quantumness thresholds. By evaluating performance at protocol level rather then the gate level, this approach provides a transparent and intuitive assessment of whether specific quantum processors, or isolated sub-chips within them, can demonstrate a practical quantum advantage. To illustrate the utility of this method, we compare two generations of IBM quantum computers: the older Eagle architecture and the newer Heron architecture. Our findings reveal the genuine operational strengths and limitations of these devices, demonstrating substantial performance improvements in the newer Heron generation.

Figures (51)

• Figure 1: The five protocols applied on the quantum computers. These protocols were defined in previous work done in this subject (ref Bench1_arxiv)
• Figure 2: Flowchart of the optimal lookup workflow. Each arrow in this flowchart has the same meaning - proceed to next stage only with sub-chips that passed the threshold in the previous stage
• Figure 3: Snapshot from the Heron r2 chip architecture. Qubits are arranged in repeating rectangular layout, we numbered each rectangle for consistent reference
• Figure 4: Transmit protocol
• Figure 5: Results of Brisbane c2c do-nothing protocol, taken four months apart. In this test all the rectangles in the quantum computer are tested, 18 rectangles for each test. The blue bars present the mean fidelity while the green and the red dots are the max and min respectively. The red dotted line mark the fidelity threshold for this protocol, in do-nothing protocol the threshold is $\frac{2}{3}$