TensorRL-QAS: Reinforcement learning with tensor networks for improved quantum architecture search

TL;DR

TensorRL-QAS tackles scalability bottlenecks in reinforcement-learning-based quantum architecture search by initializing PQCs from a DMRG-derived MPS ground state and mapping this TN to a hardware-efficient circuit via Riemannian optimization on the Stiefel manifold, followed by RL refinement. The method yields two variants (trainable TN-init and fixed TN-init) that consistently produce more compact, deeper-light PQCs while maintaining chemical accuracy for molecular systems up to 12 qubits and demonstrating robustness in noisy simulations and even larger 20-qubit Ising-type models. Across noiseless and noisy regimes, TensorRL-QAS achieves up to an order of magnitude reduction in CNOT counts, circuit depth, and classical optimizer evaluations, with up to 50% success probability for 10-qubit systems, surpassing prior RL-QAS baselines. The approach enables CPU-friendly training, scalable quantum circuit discovery, and practical applicability to near-term hardware, supported by open-source code for reproduction.

Abstract

Variational quantum algorithms hold the promise to address meaningful quantum problems already on noisy intermediate-scale quantum hardware. In spite of the promise, they face the challenge of designing quantum circuits that both solve the target problem and comply with device limitations. Quantum architecture search (QAS) automates the design process of quantum circuits, with reinforcement learning (RL) emerging as a promising approach. Yet, RL-based QAS methods encounter significant scalability issues, as computational and training costs grow rapidly with the number of qubits, circuit depth, and hardware noise. To address these challenges, we introduce $\textit{TensorRL-QAS}$, an improved framework that combines tensor network methods with RL for QAS. By warm-starting the QAS with a matrix product state approximation of the target solution, TensorRL-QAS effectively narrows the search space to physically meaningful circuits and accelerates the convergence to the desired solution. Tested on several quantum chemistry problems of up to 12-qubit, TensorRL-QAS achieves up to a 10-fold reduction in CNOT count and circuit depth compared to baseline methods, while maintaining or surpassing chemical accuracy. It reduces classical optimizer function evaluation by up to 100-fold, accelerates training episodes by up to 98$\%$, and can achieve 50$\%$ success probability for 10-qubit systems, far exceeding the $<$1$\%$ rates of baseline. Robustness and versatility are demonstrated both in the noiseless and noisy scenarios, where we report a simulation of an 8-qubit system. Furthermore, TensorRL-QAS demonstrates effectiveness on systems on 20-qubit quantum systems, positioning it as a state-of-the-art quantum circuit discovery framework for near-term hardware and beyond.

Figures (9)

• Figure 1: Schematic representation of TensorRL-QAS algorithm. Given a Hamiltonian for which we seek the lowest eigenstate. By combining tensor network (TN) with reinforcement learning (RL)-based quantum architecture search (QAS), we find solutions that would not be achievable using either approach alone. Inspired by Rudolph2023Synergistic, we obtain an approximate ground state of the Hamiltonian using DMRG WhiteDMRG1992SchollwckDMRG2011, then use this result to warm-start the QAS training in RL-framework. We employ Riemannian optimization on the Stiefel manifold to map the MPS obtained from DMRG to a quantum circuit LuchnikovReimaniannGD2021.
• Figure 2: (a) In TensorRL (trainable TN-init), an MPS is transformed into a brickwork circuit structure using Riemannian optimization. The circuit is encoded into a binary encoding to make it visible to the RL-agent. The RL-agent chooses a gate, and the information corresponding to the gate is encoded into a new binary matrix, highlighted in red. In StructureRL, we follow the same steps as above but replace the parameters with zero. (b) TensorRL (fixed TN-init) do not directly encode the MPS structure and its parameters into the RL-state, but the empty PQC is initialized with the MPS wavefunction.
• Figure 3: TensorRL (trainable) and TensorRL (fixed) require 80% and 98% less time, respectively, to execute an episode compared to CRLQAS algorithms. Meanwhile, TensorRL (trainable) and TensorRL (fixed) improve the number of function evaluations (nfev) by the classical optimizer by 10-100 fold, availing the corridor for more exploration-exploitation by the $\epsilon$-greedy agent in a fixed computational budget. The error bar (on the right figure) is the standard deviation ($\sigma$) from the average over $5$ random initializations of the neural network. The molecule in this example is 8-qubit $\texttt{H}_2\texttt{O}$.
• Figure 4: Cumulative reward vs. episodes for all evaluated methods in the task of finding the ground state of 8-$\texttt{H}_2\texttt{O}$. All methods are trained for $\sim 48$ hours, and the shaded regions represent the standard deviation across runs. TensorRL (trainable $\chi=2$) shows a sharp increase in accumulated reward compared to other methods, showing enhanced trainability. The shaded area in the plot corresponds to the standard deviation ($\sigma$) from the average over $5$ random initializations of the neural network.
• Figure 5: The probability of finding a successful episode i.e. a quantum circuit that finds the ground state of 8-, 10-$\texttt{H}_2\texttt{O}$ consistently increases with TensorRL (fixed). The error bar represents maximum and minimum values across 5 random neural network initializations, highlighting the robustness of TensorRL (fixed) even at its worst initialization compared to other methods.