A Study on Optimization Techniques for Variational Quantum Circuits in Reinforcement Learning

TL;DR

This work investigates optimization techniques for variational quantum circuits used as actors in a quantum PPO framework for reinforcement learning. By evaluating data re-uploading, input scaling, output scaling, and exponential learning-rate decay in Frozen Lake and Cart Pole environments, the authors systematically assess parameter-efficiency and learning robustness against a classical PPO baseline. Key findings show that data re-uploading and trainable output scaling substantially improve stability and learning speed, while exponential learning-rate decay further stabilizes training; however, none of the VQC-based approaches consistently outperform the classical PPO with an equivalent parameter count. The study provides practical guidance for designing VQC-based RL agents in the NISQ era and highlights directions for future work, including continuous action spaces and deployment on real quantum hardware to assess quantum-noise effects.

Abstract

Quantum Computing aims to streamline machine learning, making it more effective with fewer trainable parameters. This reduction of parameters can speed up the learning process and reduce the use of computational resources. However, in the current phase of quantum computing development, known as the noisy intermediate-scale quantum era (NISQ), learning is difficult due to a limited number of qubits and widespread quantum noise. To overcome these challenges, researchers are focusing on variational quantum circuits (VQCs). VQCs are hybrid algorithms that merge a quantum circuit, which can be adjusted through parameters, with traditional classical optimization techniques. These circuits require only few qubits for effective learning. Recent studies have presented new ways of applying VQCs to reinforcement learning, showing promising results that warrant further exploration. This study investigates the effects of various techniques -- data re-uploading, input scaling, output scaling -- and introduces exponential learning rate decay in the quantum proximal policy optimization algorithm's actor-VQC. We assess these methods in the popular Frozen Lake and Cart Pole environments. Our focus is on their ability to reduce the number of parameters in the VQC without losing effectiveness. Our findings indicate that data re-uploading and an exponential learning rate decay significantly enhance hyperparameter stability and overall performance. While input scaling does not improve parameter efficiency, output scaling effectively manages greediness, leading to increased learning speed and robustness.

Figures (9)

• Figure 1: Overview of our methodology. Green boxes denote the techniques we investigated in this work.
• Figure 2: Data re-uploading circuits. (a) For Frozen Lake: Utilizes 6 standard data re-uploading layers, each with an encoding and a variational layer featuring C-NOT entanglement. The encoding layer inputs the binary state $s$ digits, and the variational layer processes rescaled parameters $\phi = \pi \times \tanh(\theta)$, with Pauli-Z measurements on all qubits. (b) For Cart Pole: Also employs 6 data re-uploading layers, similar to the Frozen Lake circuit but differs in the last layer. The variational layer uses rescaled parameters $\phi = \pi \times \tanh(\theta)$. The encoding layer, without input scaling, inputs state dimensions $z_{RES}=rescale(s)$ or, with input scaling, the values $z_{IS}=\pi \times \tanh(\lambda \times s)$, utilizing input scaling parameters $\lambda$ and the environmental state $s$, with Pauli-Z measurements on the last two qubits.
• Figure 3: Data re-uploading circuits for Frozen Lake. (a) Following Kölle et al.Hgog2023, this circuit uses 6 data re-uploading layers with parametrized rotations and CNOT entanglement. (b) Based on Jerbi et al.QRL_jerbi2021parametrized, it employs 8 data re-uploading layers with parametrized rotations and ontrolled-Z entanglement, with a Hadamard gate and an additional variational layer preceding them.
• Figure 4: QPPO with manual re-scaling, global input scaling, input scaling (with 4 and 5 layers), and classical PPO in Cart Pole.
• Figure 5: Comparative analysis across different approaches and environments: (a) evaluates the average reward over three runs using data re-uploading, global output scaling, and exponentially decaying learning rate; (b) illustrates the impact of local output scaling in Frozen Lake allowing for a higher $lr=2.5 \times 10^{-2}$ compared to (a) $lr=5 \times 10^{-3}$; and (c) compares the standard QPPO approach with circuits from Kölle et al.Hgog2023 and Jerbi et al.QRL_jerbi2021parametrized in Frozen Lake. The number of required actor parameters is shown in parentheses, with all approaches using a critic with 5313 parameters meyer2022survey.