Hybrid Quantum-Classical Policy Gradient for Adaptive Control of Cyber-Physical Systems: A Comparative Study of VQC vs. MLP

TL;DR

The paper compares classical MLP and quantum VQC policies for reinforcement learning in cyber-physical control, using a unified framework with $J(\theta)=\mathbb{E}_{\pi_\theta}[\sum_t \gamma^t r_t]$ and REINFORCE with a baseline. It demonstrates that the classical MLP rapidly achieves near-optimal performance on CartPole-v1 ($\bar{J} \approx 498.7$) while the VQC remains limited by expressivity of a shallow, four-qubit circuit, though it benefits from far fewer parameters and smoother optimization via the parameter-shift gradient. Under Gaussian observation noise, the MLP degrades gracefully whereas the VQC shows strong sensitivity, highlighting expressivity constraints of current quantum policies. The study concludes that hybrid quantum-classical RL holds promise for efficiency and robustness as hardware noise and circuit depth improve, warranting deeper entangling architectures and noise-mitigation techniques for practical CPS deployment.

Abstract

The comparative evaluation between classical and quantum reinforcement learning (QRL) paradigms was conducted to investigate their convergence behavior, robustness under observational noise, and computational efficiency in a benchmark control environment. The study employed a multilayer perceptron (MLP) agent as a classical baseline and a parameterized variational quantum circuit (VQC) as a quantum counterpart, both trained on the CartPole-v1 environment over 500 episodes. Empirical results demonstrated that the classical MLP achieved near-optimal policy convergence with a mean return of 498.7 +/- 3.2, maintaining stable equilibrium throughout training. In contrast, the VQC exhibited limited learning capability, with an average return of 14.6 +/- 4.8, primarily constrained by circuit depth and qubit connectivity. Noise robustness analysis further revealed that the MLP policy deteriorated gracefully under Gaussian perturbations, while the VQC displayed higher sensitivity at equivalent noise levels. Despite the lower asymptotic performance, the VQC exhibited significantly lower parameter count and marginally increased training time, highlighting its potential scalability for low-resource quantum processors. The results suggest that while classical neural policies remain dominant in current control benchmarks, quantum-enhanced architectures could offer promising efficiency advantages once hardware noise and expressivity limitations are mitigated.

Figures (5)

• Figure 1: Framework of Hybrid Quantum-Classical Policy Gradient Reinforcement Learning to integrate a Variational Quantum Circuit (VQC) and a Classical MLP.
• Figure 2: Learning curves of classical (MLP) and quantum (VQC) agents over 400 training episodes in the CartPole-v1 environment. Both raw and smoothed (MA(10)) returns are displayed.
• Figure 3: Convergence stability comparison (zoom-in view) between MLP and VQC policies over the last 100 episodes, showing 10-episode moving averages.
• Figure 4: Average episodic return under varying observation noise levels ($\sigma$). The MLP exhibits graceful degradation, while the VQC remains near the baseline due to under-parameterization.
• Figure 5: Computational efficiency of MLP and VQC agents in terms of parameter count and wall-clock training time.