Fair Benchmarking of Optimisation Applications

TL;DR

The paper tackles fair benchmarking of quantum optimisation, where end-to-end evaluation is needed to avoid overstating quantum advantage. It proposes a principled framework of guidelines and practical protocols to measure performance across diverse paradigms, including $QAOA$ and $VQE$, with attention to end-to-end timing, tuning disclosure, comparable algorithm classes, problem diversity, and energy-aware metrics. It builds on prior work such as TAQOS, Q-Score, Qoptlib, and QEI to operationalize baselines, workflow costs, and sustainability considerations. By capturing full hybrid quantum–classical workloads, the approach aims for reproducibility and trustworthy comparisons that reflect real-world use cases. The framework is intended to guide researchers, industry practitioners, and policymakers toward responsible deployment of quantum optimisation technologies.

Abstract

Quantum optimisation is emerging as a promising approach alongside classical heuristics and specialised hardware, yet its performance is often difficult to assess fairly. Traditional benchmarking methods, rooted in digital complexity theory, do not directly capture the continuous dynamics, probabilistic outcomes, and workflow overheads of quantum and hybrid systems. This paper proposes principles and protocols for fair benchmarking of quantum optimisation, emphasising end-to-end workflows, transparency in tuning and reporting, problem diversity, and avoidance of speculative claims. By extending lessons from classical benchmarking and incorporating application-driven and energy-aware metrics, we outline a framework that enables practitioners to evaluate quantum methods responsibly, ensuring reproducibility, comparability, and trust in reported results.