HamilToniQ: An Open-Source Benchmark Toolkit for Quantum Computers

TL;DR

HamilToniQ addresses the lack of standardized, application-oriented benchmarks for quantum processors by offering an open-source toolkit that benchmarks QPUs via QAOA-based tasks and a standardized H-Score that integrates hardware, compilation, and error-mitigation considerations. It uses a ground-truth noiseless reference and a PDF-based scoring framework, enabling self-normalized, cross-system comparisons and support for distributed workloads in Quantum-HPC environments. The authors validate the approach on IBM QPUs, examining topology, error-mitigation protocols, and the benefits of optimized compilation for multi-QPU scenarios. The work provides a scalable, transparent benchmark framework for the full quantum software stack and practical guidance for resource management in future quantum infrastructures.

Abstract

In this paper, we introduce HamilToniQ, an open-source, and application-oriented benchmarking toolkit for the comprehensive evaluation of Quantum Processing Units (QPUs). Designed to navigate the complexities of quantum computations, HamilToniQ incorporates a methodological framework assessing QPU types, topologies, and multi-QPU systems. The toolkit facilitates the evaluation of QPUs' performance through multiple steps including quantum circuit compilation and quantum error mitigation (QEM), integrating strategies that are unique to each stage. HamilToniQ's standardized score, H-Score, quantifies the fidelity and reliability of QPUs, providing a multidimensional perspective of QPU performance. With a focus on the Quantum Approximate Optimization Algorithm (QAOA), the toolkit enables direct, comparable analysis of QPUs, enhancing transparency and equity in benchmarking. Demonstrated in this paper, HamilToniQ has been validated on various IBM QPUs, affirming its effectiveness and robustness. Overall, HamilToniQ significantly contributes to the advancement of the quantum computing field by offering precise and equitable benchmarking metrics.

Figures (14)

• Figure 1: Compilation Process of a QAOA Algorithm.
• Figure 2: Flowchart of HamilToniQ Benchmarking Software Workflow: Demonstrating the integration of QPU characteristics, compilation and error mitigation strategies, and their effects on benchmarking outcomes for enhanced reliability (H-Score) and execution time.
• Figure 3: This flow chart illustrates the standard benchmarking procedures of a QPU using HamilToniQ. Users can use the Q matrices and scoring curves provided within hamiltoniq.instances. Alternatively, the Toniq.get_reference function in hamiltoniq.benchmark can calculate the scoring curves of users' own Q matrices. The performance depends on the accuracy of solving QAOA problems using each QPU.
• Figure 4: An example illustrates the distributions of accuracy (solid lines) and their cumulative sum (dashed lines). The cumulative sum is also the scoring curve mentioned in Sec. \ref{['Ch: reference']}. Two configurations of the QAOA were used: one with a single layer (blue lines) and another with 9 layers (purple lines). It is apparent that QAOA with 9 layers performs better than the other one, as more results have higher accuracy. This conclusion shows the shift of the peak of accuracy distribution towards the right in the figure. The results were obtained from the qubit_3 instance given in hamiltoniq.matrices.
• Figure 5: The accuracy distributions are illustrated for (a) a noiseless simulator using 3 qubits, (b) ibm_lagos using 3 qubits, and (c) ibm_cairo using 3 qubits. The vertical axis indicates the number of layers and the color represents the probability density. The brighter the color, the higher the probability density. In (a), the distribution for 1 layer and 9 layers correspond to the blue solid line and purple solid line in Fig. \ref{['fig: pdf_scoring']}, respectively. These results were obtained on qubit_3 instances given in hamiltoniq.instances.