Benchmarking the Operation of Quantum Heuristics and Ising Machines: Scoring Parameter Setting Strategies on Optimization Applications

TL;DR

This work tackles performance benchmarking for parameterized stochastic solvers used in quantum and quantum-inspired optimization, where outcomes are distributed rather than deterministic and are characterized by a real-valued metric $X = \mathrm{fun}(\mathbf{z})$. It introduces an operational benchmarking framework and an open-source toolkit, Stochastic-Benchmark, that treats solvers as parameterized samplers and explicitly accounts for tuning overhead and instance variability across problem distributions. Key contributions include the virtual best performance profile, fixed and adaptive parameter-setting strategies, Hyperopt-based search with exploration–exploitation control, and cross-validated window-sticker visualizations, demonstrated on a CIM-CAC and parallel tempering with Wishart planted instances. This framework enables realistic, resource-aware, reproducible comparisons of hybrid quantum-classical and quantum-inspired solvers across diverse hardware platforms and problem classes.

Abstract

We discuss guidelines for evaluating the performance of parameterized stochastic solvers for optimization problems, with particular attention to systems that employ novel hardware, such as digital quantum processors running variational algorithms, analog processors performing quantum annealing, or coherent Ising Machines. We illustrate through an example a benchmarking procedure grounded in the statistical analysis of the expectation of a given performance metric measured in a test environment. In particular, we discuss the necessity and cost of setting parameters that affect the algorithm's performance. The optimal value of these parameters could vary significantly between instances of the same target problem. We present an open-source software package that facilitates the design, evaluation, and visualization of practical parameter tuning strategies for complex use of the heterogeneous components of the solver. We examine in detail an example using parallel tempering and a simulator of a photonic Coherent Ising Machine computing and display the scoring of an illustrative baseline family of parameter-setting strategies that feature an exploration-exploitation trade-off.

Figures (6)

• Figure 1: Abstract conceptualization of a solution method and a solver. The black box indicates the core processing optimizer (e.g., a quantum device) primarily responsible for the method performance.
• Figure 2: Flowchart with the main steps to generate the "Window stickers" implemented in Stochastic-BenchmarkE_Bernal_Neira_Stochastic_Benchmark_toolkit_2023
• Figure 3: Cross-validated performance profiles from 10 test-train splits of 50 Wishart instances with $N=50$ and $\alpha=0.5$ solved via (left) CIM-CAC Chencimoptimizer2022 and (right) PySA Mandra_PySA_Fast_Simulated_2023. The profiles of the virtual best baseline, a Hyperopt-driven exploration-exploitation strategy, and the fixed best parameters suggested from the experiments are shown. (generated by Stochastic-Benchmark)
• Figure 4: Parameter strategy plots applied to (left) CIM-CAC Chencimoptimizer2022 and (right) PySA Mandra_PySA_Fast_Simulated_2023. Same instances and legends as Fig. \ref{['fig:performance_CIM_and_PySA']}
• Figure 5: Meta-parameter strategy plots for exploration-exploitation strategy, applied to (left) the CIM-CAC Chencimoptimizer2022 and (right) PySA Mandra_PySA_Fast_Simulated_2023. For CIM-CAC, the meta-parameter $\tau = 1$ for all resources probed. The dashed line represents those meta-parameters with best-found performance, and the continuous line represents the actionable implementation. Same instances and legends as Fig. \ref{['fig:performance_CIM_and_PySA']}.(generated by Stochastic-Benchmark)