Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QuanBench: Benchmarking Quantum Code Generation with Large Language Models

Xiaoyu Guo, Minggu Wang, Jianjun Zhao

TL;DR

QuanBench introduces a dedicated benchmark to evaluate LLMs on generating functionally correct and semantically faithful quantum programs, addressing a gap in quantum software development. It assembles 44 tasks across four categories with a dual evaluation regime (Pass@K and Process Fidelity) and evaluates nine state-of-the-art LLMs, finding limited capability in producing correct quantum code (Pass@1<0.4, Pass@5~0.5) and average semantic fidelity around 0.5, with notable semantic and structural error patterns. The study reveals that performance does not strictly track model size and that deeper quantum reasoning, such as operator composition and entanglement, remains challenging, suggesting a need for domain-adapted training and improved reasoning. It also analyzes error modes (deprecated APIs, circuit construction, semantic deviations) and offers guidelines on temperature and batch-size settings for reliable benchmarking. QuanBench thus provides a foundation for systematic, task-diverse quantum-code evaluation and points to future work including multi-framework support and retrieval-augmented generation to enhance quantum programming capabilities.

Abstract

Large language models (LLMs) have demonstrated good performance in general code generation; however, their capabilities in quantum code generation remain insufficiently studied. This paper presents QuanBench, a benchmark for evaluating LLMs on quantum code generation. QuanBench includes 44 programming tasks that cover quantum algorithms, state preparation, gate decomposition, and quantum machine learning. Each task has an executable canonical solution and is evaluated by functional correctness (Pass@K) and quantum semantic equivalence (Process Fidelity). We evaluate several recent LLMs, including general-purpose and code-specialized models. The results show that current LLMs have limited capability in generating the correct quantum code, with overall accuracy below 40% and frequent semantic errors. We also analyze common failure cases, such as outdated API usage, circuit construction errors, and incorrect algorithm logic. QuanBench provides a basis for future work on improving quantum code generation with LLMs.

QuanBench: Benchmarking Quantum Code Generation with Large Language Models

TL;DR

QuanBench introduces a dedicated benchmark to evaluate LLMs on generating functionally correct and semantically faithful quantum programs, addressing a gap in quantum software development. It assembles 44 tasks across four categories with a dual evaluation regime (Pass@K and Process Fidelity) and evaluates nine state-of-the-art LLMs, finding limited capability in producing correct quantum code (Pass@1<0.4, Pass@5~0.5) and average semantic fidelity around 0.5, with notable semantic and structural error patterns. The study reveals that performance does not strictly track model size and that deeper quantum reasoning, such as operator composition and entanglement, remains challenging, suggesting a need for domain-adapted training and improved reasoning. It also analyzes error modes (deprecated APIs, circuit construction, semantic deviations) and offers guidelines on temperature and batch-size settings for reliable benchmarking. QuanBench thus provides a foundation for systematic, task-diverse quantum-code evaluation and points to future work including multi-framework support and retrieval-augmented generation to enhance quantum programming capabilities.

Abstract

Large language models (LLMs) have demonstrated good performance in general code generation; however, their capabilities in quantum code generation remain insufficiently studied. This paper presents QuanBench, a benchmark for evaluating LLMs on quantum code generation. QuanBench includes 44 programming tasks that cover quantum algorithms, state preparation, gate decomposition, and quantum machine learning. Each task has an executable canonical solution and is evaluated by functional correctness (Pass@K) and quantum semantic equivalence (Process Fidelity). We evaluate several recent LLMs, including general-purpose and code-specialized models. The results show that current LLMs have limited capability in generating the correct quantum code, with overall accuracy below 40% and frequent semantic errors. We also analyze common failure cases, such as outdated API usage, circuit construction errors, and incorrect algorithm logic. QuanBench provides a basis for future work on improving quantum code generation with LLMs.
Paper Structure (29 sections, 4 equations, 8 figures, 6 tables)

This paper contains 29 sections, 4 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Large Language Models
  4. Benchmarks for Code Generation
  5. Quantum Computing
  6. Benchmark Construction
  7. Problem Selection
  8. Rewriting Problems and Canonical Solutions
  9. Docstring Construction
  10. Implementing Multi-Criteria Evaluations
  11. Dataset Statistics
  12. Dataset Expansion
  13. Evaluation of LLM Models
  14. Research Questions (RQs)
  15. Model Selection and Settings
  16. ...and 14 more sections

Figures (8)

  • Figure 1: QuanBench Design
  • Figure 2: One Example in QuanBench
  • Figure 3: Effectiveness comparison between LLMs on QuanBench
  • Figure 4: Number of problems solved on QuanBench
  • Figure 5: Different Constructions of GHZ State
  • ...and 3 more figures