Reinforcement Learning for Parameterized Quantum State Preparation: A Comparative Study

TL;DR

The paper addresses the challenge of parameterized quantum state preparation within directed quantum circuit synthesis by integrating continuous rotation angles into a reinforcement-learning loop. It compares a one-stage policy that jointly selects gates, qubits, and rotation angles against a two-stage approach that first designs a discrete circuit and then optimizes angles with gradient-based methods. Using PPO and A2C in a Gymnasium–PennyLane environment across 2–10 qubits and depth budgets $L=2\\lambda$ (with $\\lambda\in\\{1,...,5\\}$), the study finds that A2C fails while PPO yields stable improvements; the one-stage policy emerges as the practical choice, achieving high fidelity on basis and Bell states but with scalability limits around $\\lambda\leq3\\text{--}4$ and difficulties at ten qubits. The two-stage method offers at most marginal gains at roughly three times the runtime and cannot rectify poor discrete topologies, motivating its recommendation only when additional computational resources justify its use. The results establish a baseline for continuous-parameter RL in quantum control and highlight avenues for improvement (e.g., curriculum design, initialization, expanded gate sets) to push scalability further toward larger, more entangled states.

Abstract

We extend directed quantum circuit synthesis (DQCS) with reinforcement learning from purely discrete gate selection to parameterized quantum state preparation with continuous single-qubit rotations \(R_x\), \(R_y\), and \(R_z\). We compare two training regimes: a one-stage agent that jointly selects the gate type, the affected qubit(s), and the rotation angle; and a two-stage variant that first proposes a discrete circuit and subsequently optimizes the rotation angles with Adam using parameter-shift gradients. Using Gymnasium and PennyLane, we evaluate Proximal Policy Optimization (PPO) and Advantage Actor--Critic (A2C) on systems comprising two to ten qubits and on targets of increasing complexity with \(λ\) ranging from one to five. Whereas A2C does not learn effective policies in this setting, PPO succeeds under stable hyperparameters (one-stage: learning rate approximately \(5\times10^{-4}\) with a self-fidelity-error threshold of 0.01; two-stage: learning rate approximately \(10^{-4}\)). Both approaches reliably reconstruct computational basis states (between 83\% and 99\% success) and Bell states (between 61\% and 77\% success). However, scalability saturates for \(λ\) of approximately three to four and does not extend to ten-qubit targets even at \(λ=2\). The two-stage method offers only marginal accuracy gains while requiring around three times the runtime. For practicality under a fixed compute budget, we therefore recommend the one-stage PPO policy, provide explicit synthesized circuits, and contrast with a classical variational baseline to outline avenues for improved scalability.

Figures (3)

• Figure 1: Hybrid loop: Gymnasium manages the RL interface, PennyLane executes the quantum circuit, and the policy samples discrete gates, qubits, and continuous angles.
• Figure 2: DQCS landscape across target complexity $\lambda$ and qubit counts. The two-stage strategy occasionally improves fidelity but requires around three times the runtime. Mean values in solid lines, 95% confidence intervals in shaded regions.
• Figure 3: Example circuits synthesized by the one-stage agent. Rotation gates align single-qubit phases; $\mathrm{CNOT}$ gates propagate amplitude correlations.