Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback

Taku Mikuriya, Tatsuya Ishigaki, Masayuki Kawarada, Shunya Minami, Tadashi Kadowaki, Yohichi Suzuki, Soshun Naito, Shunya Takata, Takumi Kato, Tamotsu Basseda, Reo Yamada, Hiroya Takamura

TL;DR

QCoder Benchmark introduces a hardware-aware evaluation framework for quantum code generation that combines a dataset of quantum problems with multiple human-solutions and a simulator-based evaluator providing domain-specific feedback. Baseline prompting paired with iterative, feedback-driven refinement substantially improves code quality, revealing that reasoning-focused models like o3 outperform both other LLMs and average human submissions in this constrained domain. The study demonstrates the value of integrating hardware feedback into prompts to surpass conventional functional correctness alone, and provides public resources to catalyze further research in hardware-constrained code generation. The approach shows promise for extending to robotics and embedded systems where execution constraints are as important as correctness.

Abstract

Large language models (LLMs) have increasingly been applied to automatic programming code generation. This task can be viewed as a language generation task that bridges natural language, human knowledge, and programming logic. However, it remains underexplored in domains that require interaction with hardware devices, such as quantum programming, where human coders write Python code that is executed on a quantum computer. To address this gap, we introduce QCoder Benchmark, an evaluation framework that assesses LLMs on quantum programming with feedback from simulated hardware devices. Our benchmark offers two key features. First, it supports evaluation using a quantum simulator environment beyond conventional Python execution, allowing feedback of domain-specific metrics such as circuit depth, execution time, and error classification, which can be used to guide better generation. Second, it incorporates human-written code submissions collected from real programming contests, enabling both quantitative comparisons and qualitative analyses of LLM outputs against human-written codes. Our experiments reveal that even advanced models like GPT-4o achieve only around 18.97% accuracy, highlighting the difficulty of the benchmark. In contrast, reasoning-based models such as o3 reach up to 78% accuracy, outperforming averaged success rates of human-written codes (39.98%). We release the QCoder Benchmark dataset and public evaluation API to support further research. (Codes and datasets are available at https://qcoder-bench.github.io/ )

QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback

TL;DR

QCoder Benchmark introduces a hardware-aware evaluation framework for quantum code generation that combines a dataset of quantum problems with multiple human-solutions and a simulator-based evaluator providing domain-specific feedback. Baseline prompting paired with iterative, feedback-driven refinement substantially improves code quality, revealing that reasoning-focused models like o3 outperform both other LLMs and average human submissions in this constrained domain. The study demonstrates the value of integrating hardware feedback into prompts to surpass conventional functional correctness alone, and provides public resources to catalyze further research in hardware-constrained code generation. The approach shows promise for extending to robotics and embedded systems where execution constraints are as important as correctness.

Abstract

Large language models (LLMs) have increasingly been applied to automatic programming code generation. This task can be viewed as a language generation task that bridges natural language, human knowledge, and programming logic. However, it remains underexplored in domains that require interaction with hardware devices, such as quantum programming, where human coders write Python code that is executed on a quantum computer. To address this gap, we introduce QCoder Benchmark, an evaluation framework that assesses LLMs on quantum programming with feedback from simulated hardware devices. Our benchmark offers two key features. First, it supports evaluation using a quantum simulator environment beyond conventional Python execution, allowing feedback of domain-specific metrics such as circuit depth, execution time, and error classification, which can be used to guide better generation. Second, it incorporates human-written code submissions collected from real programming contests, enabling both quantitative comparisons and qualitative analyses of LLM outputs against human-written codes. Our experiments reveal that even advanced models like GPT-4o achieve only around 18.97% accuracy, highlighting the difficulty of the benchmark. In contrast, reasoning-based models such as o3 reach up to 78% accuracy, outperforming averaged success rates of human-written codes (39.98%). We release the QCoder Benchmark dataset and public evaluation API to support further research. (Codes and datasets are available at https://qcoder-bench.github.io/ )

Paper Structure

This paper contains 19 sections, 3 equations, 11 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. QCoder Benchmark
  4. Dataset
  5. Simulator-based Evaluator
  6. Statistics and Comparisons with Other Datasets
  7. Methods
  8. Baseline Prompting Strategy
  9. Feedback-aware Code Refinement
  10. Experiments
  11. Compared LLMs
  12. Evaluation Metrics
  13. Success rate.
  14. Fine-grained Failure Rates.
  15. Success Rates of Human Submitted Codes
  16. ...and 4 more sections

Figures (11)

  • Figure 1: Quantum code generation involves generating a python code that constructs a quantum circuit executable on a quantum computer. Due to strict constraints of actual hardware, feedback from the hardware is necessary to generate executable codes on a quantum computer.
  • Figure 2: Example prompt for quantum code generation.
  • Figure 3: Example of the baseline prompt.
  • Figure 4: The prompt used for iterative refinement.
  • Figure 5: The changes of success rate when we change the number of refinement iterations.
  • ...and 6 more figures