Empirical Studies on Quantum Optimization for Software Engineering: A Systematic Analysis

TL;DR

This paper addresses the need for rigorous empirical evaluation practices in quantum optimization for software engineering by systematically analyzing 76 primary studies from a recent SLR. It maps evaluation designs, hyperparameters, experimental settings, metrics, case studies, baselines, and artifact availability, revealing gaps in reporting (repetitions, shots, noise), lack of standardized metrics, and limited open artifacts. Key contributions include a detailed characterization of current practices, identification of gaps, and practical guidelines to strengthen future empirical studies, including advocating for open-source artifacts and standardized reporting of resource costs and experimental settings. The findings have practical impact by guiding researchers and practitioners toward fairer, more reproducible, and cost-aware assessments of quantum, quantum-inspired, and hybrid SE optimization approaches. Overall, this work lays an foundational reference for designing and conducting robust empirical evaluations in this emerging interdisciplinary area.

Abstract

In recent years, quantum, quantum-inspired, and hybrid algorithms are increasingly showing promise for solving software engineering optimization problems. However, best-intended practices for conducting empirical studies have not yet well established. In this paper, based on the primary studies identified from the latest systematic literature review on quantum optimization for software engineering problems, we conducted a systematic analysis on these studies from various aspects including experimental designs, hyperparameter settings, case studies, baselines, tooling, and metrics. We identify key gaps in the current practices such as limited reporting of the number of repetitions, number of shots, and inadequate consideration of noise handling, as well as a lack of standardized evaluation protocols such as the adoption of quality metrics, especially quantum-specific metrics. Based on our analysis, we provide insights for designing empirical studies and highlight the need for more real-world and open case studies to assess cost-effectiveness and practical utility of the three types of approaches: quantum-inspired, quantum, and hybrid. This study is intended to offer an overview of current practices and serve as an initial reference for designing and conducting empirical studies on evaluating and comparing quantum, quantum-inspired, and hybrid algorithms in solving optimization problems in software engineering.

Figures (11)

• Figure 1: Overview of evaluation design (a) and evaluation metrics (b) - RQ1
• Figure 2: Overview of parameter settings - RQ3
• Figure 3: Overview of settings of repetitions - RQ3
• Figure 4: Overview of settings of shots - RQ3
• Figure 5: Overview of the primary studies that specify the number of qubits - RQ3