Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Quantum Computers: Towards a Standard Performance Evaluation Approach

Arturo Acuaviva, David Aguirre, Rubén Peña, Mikel Sanz

TL;DR

This paper analyzes how classical benchmarking principles can inform fair quantum processor evaluation. It defines a unified quantum benchmarking vocabulary, surveys existing techniques, metrics, and NISQ-era benchmarks, and proposes general guidelines and a roadmap toward standardization. A central contribution is the SPEQC framework, a flexible, multi-era organization designed to develop and promote standardized benchmarks and reporting for quantum devices. The work aims to reduce bias, improve comparability across platforms, and steer quantum hardware development toward robust, scalable performance evaluation.

Abstract

The technological development of increasingly larger quantum processors on different quantum platforms raises the problem of how to fairly compare their performance, known as quantum benchmarking of quantum processors. This is a challenge that computer scientists have already faced when comparing classical processors, leading to the development of various mathematical tools to address it, but also to the identification of the limits of this problem. In this work, we briefly review the most important aspects of both classical processor benchmarks and the metrics comprising them, providing precise definitions and analyzing the quality attributes that they should exhibit. Subsequently, we analyze the intrinsic properties that characterize the paradigm of quantum computing and hinder the naive transfer of strategies from classical benchmarking. However, we can still leverage some of the lessons learned such as the quality attributes of a \textit{good} benchmark. Additionally, we review some of the most important metrics and benchmarks for quantum processors proposed in the literature, assessing what quality attributes they fulfill. Finally, we propose general guidelines for quantum benchmarking. These guidelines aim to pave the way for establishing a roadmap towards standardizing the performance evaluation of quantum devices, ultimately leading to the creation of an organization akin to the Standard Performance Evaluation Corporation (SPEC).

Benchmarking Quantum Computers: Towards a Standard Performance Evaluation Approach

TL;DR

This paper analyzes how classical benchmarking principles can inform fair quantum processor evaluation. It defines a unified quantum benchmarking vocabulary, surveys existing techniques, metrics, and NISQ-era benchmarks, and proposes general guidelines and a roadmap toward standardization. A central contribution is the SPEQC framework, a flexible, multi-era organization designed to develop and promote standardized benchmarks and reporting for quantum devices. The work aims to reduce bias, improve comparability across platforms, and steer quantum hardware development toward robust, scalable performance evaluation.

Abstract

The technological development of increasingly larger quantum processors on different quantum platforms raises the problem of how to fairly compare their performance, known as quantum benchmarking of quantum processors. This is a challenge that computer scientists have already faced when comparing classical processors, leading to the development of various mathematical tools to address it, but also to the identification of the limits of this problem. In this work, we briefly review the most important aspects of both classical processor benchmarks and the metrics comprising them, providing precise definitions and analyzing the quality attributes that they should exhibit. Subsequently, we analyze the intrinsic properties that characterize the paradigm of quantum computing and hinder the naive transfer of strategies from classical benchmarking. However, we can still leverage some of the lessons learned such as the quality attributes of a \textit{good} benchmark. Additionally, we review some of the most important metrics and benchmarks for quantum processors proposed in the literature, assessing what quality attributes they fulfill. Finally, we propose general guidelines for quantum benchmarking. These guidelines aim to pave the way for establishing a roadmap towards standardizing the performance evaluation of quantum devices, ultimately leading to the creation of an organization akin to the Standard Performance Evaluation Corporation (SPEC).
Paper Structure (19 sections, 8 equations, 3 figures, 1 table)

This paper contains 19 sections, 8 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. What can we learn from Classical benchmarking?
  3. Characteristics of a good benchmark
  4. What types of benchmarks are there?
  5. What makes a good metric
  6. Examples of classical benchmarks
  7. Quantum benchmarking
  8. Nuances of benchmarking quantum computers
  9. A common language to discuss
  10. Reviewing existing proposals for benchmarking
  11. Techniques
  12. Metrics
  13. Benchmarks in NISQ Era
  14. Roadmap for Quantum Benchmarking
  15. Guidelines for the benchmark practitioner
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Input-Processing-Output (IPO) model. IPO model can be applied to many quantum algorithms, with quantum state preparation as input, quantum information processing as data processing, and quantum state tomography as information retrieval.
  • Figure 2: Quantum benchmarking structure. We introduce a step-wise structure to benchmarking, from pre-benchmark steps for verification of the device, defining the benchmarking and executing it, up to the reporting stage.
  • Figure 3: Example of SPEQC report. Report inspired by the https://www.spec.org/cpu2017/, none of the metrics are reflecting actual measurements of the devices or real properties, their usage is limited to exemplify how the report would look. More details can be added to the report file in the form of sections below Submit Notes.

Theorems & Definitions (9)

  • Definition 1: Classical Benchmark
  • Definition 2: Performance Metric
  • Definition 3: Quantum Benchmark
  • Definition 4: Quantum Performance Metric
  • Definition 5: Quantum Verification
  • Definition 6: Quantum verification technique
  • Definition 7: Quantum Testing
  • Definition 8: Quantum Benchmark Framework/Family
  • Definition 9: Quantum Benchmark Suite