Towards Robust Benchmarking of Quantum Optimization Algorithms

TL;DR

The paper addresses the challenge of fairly benchmarking quantum optimization algorithms against classical approaches for industrial COPs. It proposes a pragmatic, use-case driven framework that emphasizes matching problem formulations, diverse and realistic datasets, holistic figures of merit (such as Time-To-Solution and Best-Solution Found), and equitable hyperparameter tuning, including VQC parameter optimization. The authors demonstrate the guidelines on Max-Cut and TSP through two MC scenarios and a QAOA-focused TSP study, showing that classical solvers can outperform quantum approaches in certain regimes, while quantum designs (notably low-depth XY-Mixer QAOA) can offer advantages in others. The work provides actionable best practices for fair comparisons, highlights the importance of problem encoding choices, and offers practical insights into how to build meaningful benchmarks that reflect real-world utility of quantum optimization. Overall, the framework facilitates more reliable assessments of quantum advantage in optimization and supports better decision-making for industry adoption.

Abstract

Benchmarking the performance of quantum optimization algorithms is crucial for identifying utility for industry-relevant use cases. Benchmarking processes vary between optimization applications and depend on user-specified goals. The heuristic nature of quantum algorithms poses challenges, especially when comparing to classical counterparts. A key problem in existing benchmarking frameworks is the lack of equal effort in optimizing for the best quantum and, respectively, classical approaches. This paper presents a comprehensive set of guidelines comprising universal steps towards fair benchmarks. We discuss (1) application-specific algorithm choice, ensuring every solver is provided with the most fitting mathematical formulation of a problem; (2) the selection of benchmark data, including hard instances and real-world samples; (3) the choice of a suitable holistic figure of merit, like time-to-solution or solution quality within time constraints; and (4) equitable hyperparameter training to eliminate bias towards a particular method. The proposed guidelines are tested across three benchmarking scenarios, utilizing the Max-Cut (MC) and Travelling Salesperson Problem (TSP). The benchmarks employ classical mathematical algorithms, such as Branch-and-Cut (BNC) solvers, classical heuristics, Quantum Annealing (QA), and the Quantum Approximate Optimization Algorithm (QAOA).

Figures (7)

• Figure 1: Overview of the benchmarking guidelines devised throughout this paper.
• Figure 2: TTS with and without overhead for different Max-Cut solvers on problems. Error bars indicate the 75% percentile interval. QAOA is not present in the TTS$_\text{oh}$ because of the simulation; a CNOT layer is estimated to take 1$\upmu\mathrm{s}$. For the sampling only TTS, we only consider heuristics.
• Figure 3: AR from all sizes for heuristic MC solvers. TS finds the optimal solution in almost every sample. It is important to note that the runtimes for estimating AR are vastly different.
• Figure 4: Max-Cut QAOA TTS in terms of CNOT layers for different QAOA rounds $p = 2,\dots,32$. A U-shape is observable for every problem size, indicating an optimal QAOA depth.
• Figure 5: Relative error in cut size within a group. The time limit is set to 10 s.