Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Systematic benchmarking of quantum computers: status and recommendations

Jeanette Miriam Lorenz, Thomas Monz, Jens Eisert, Daniel Reitzner, Félicien Schopfer, Frédéric Barbaresco, Krzysztof Kurowski, Ward van der Schoot, Thomas Strohm, Jean Senellart, Cécile M. Perrault, Martin Knufinke, Ziyad Amodjee, Mattia Giardini

TL;DR

This paper addresses the need for systematic benchmarking of quantum computers across hardware, software, and applications. It proposes a framework for defining benchmarks, selecting metrics, and aggregating results, while surveying the state-of-the-art benchmarks and ongoing standardization efforts. It emphasizes scalable, hardware-agnostic benchmarks complemented by domain-specific, hardware-dependent tests, and highlights the crucial role of standardization and HPC/cloud integration. The work aims to align diverse benchmarking initiatives into a cohesive, internationally recognized set of protocols and KPIs to accelerate progress toward useful quantum computing.

Abstract

Architectures for quantum computing can only be scaled up when they are accompanied by suitable benchmarking techniques. The document provides a comprehensive overview of the state and recommendations for systematic benchmarking of quantum computers. Benchmarking is crucial for assessing the performance of quantum computers, including the hardware, software, as well as algorithms and applications. The document highlights key aspects such as component-level, system-level, software-level, HPC-level, and application-level benchmarks. Component-level benchmarks focus on the performance of individual qubits and gates, while system-level benchmarks evaluate the entire quantum processor. Software-level benchmarks consider the compiler's efficiency and error mitigation techniques. HPC-level and cloud benchmarks address integration with classical systems and cloud platforms, respectively. Application-level benchmarks measure performance in real-world use cases. The document also discusses the importance of standardization to ensure reproducibility and comparability of benchmarks, and highlights ongoing efforts in the quantum computing community towards establishing these benchmarks. Recommendations for future steps emphasize the need for developing standardized evaluation routines and integrating benchmarks with broader quantum technology activities.

Systematic benchmarking of quantum computers: status and recommendations

TL;DR

This paper addresses the need for systematic benchmarking of quantum computers across hardware, software, and applications. It proposes a framework for defining benchmarks, selecting metrics, and aggregating results, while surveying the state-of-the-art benchmarks and ongoing standardization efforts. It emphasizes scalable, hardware-agnostic benchmarks complemented by domain-specific, hardware-dependent tests, and highlights the crucial role of standardization and HPC/cloud integration. The work aims to align diverse benchmarking initiatives into a cohesive, internationally recognized set of protocols and KPIs to accelerate progress toward useful quantum computing.

Abstract

Architectures for quantum computing can only be scaled up when they are accompanied by suitable benchmarking techniques. The document provides a comprehensive overview of the state and recommendations for systematic benchmarking of quantum computers. Benchmarking is crucial for assessing the performance of quantum computers, including the hardware, software, as well as algorithms and applications. The document highlights key aspects such as component-level, system-level, software-level, HPC-level, and application-level benchmarks. Component-level benchmarks focus on the performance of individual qubits and gates, while system-level benchmarks evaluate the entire quantum processor. Software-level benchmarks consider the compiler's efficiency and error mitigation techniques. HPC-level and cloud benchmarks address integration with classical systems and cloud platforms, respectively. Application-level benchmarks measure performance in real-world use cases. The document also discusses the importance of standardization to ensure reproducibility and comparability of benchmarks, and highlights ongoing efforts in the quantum computing community towards establishing these benchmarks. Recommendations for future steps emphasize the need for developing standardized evaluation routines and integrating benchmarks with broader quantum technology activities.

Paper Structure

This paper contains 17 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Terminology
  3. State of the art of quantum computing benchmarking world-wide
  4. How to define quantum benchmarks
  5. Benchmark versus metric
  6. Objectives for benchmarks
  7. Procedure to define a benchmark or a protocol
  8. Aggregating different metrics
  9. Different benchmarks
  10. Component-level benchmarks
  11. System-level benchmark
  12. Software-level benchmarks
  13. HPC-level and cloud benchmarks
  14. Application-level benchmarks
  15. Additional classical metrics to consider
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Benchmarks at different levels -- in addition to the typical distinction between a component-level, system-level and application-level, we have identified also the distinction into software-level benchmarks, HPC and cloud-level benchmarks, as well as specific classical metrics. Some benchmarks, especially in the system levels are not scalable. These have been shaded out. Some benchmarks in component or system level, marked by a star, are FTQC-ready and might be useful to compare over time in both regimes, NISQ as well as FTQC. This holds also for all benchmarks in the application level, but for clarity the stars are omitted.
  • Figure 2: Objectives for aggregations of metrics into an overarching metric.