A Mutual Information-based Metric for Temporal Expressivity and Trainability Estimation in Quantum Policy Gradient Pipelines

TL;DR

The paper tackles the challenge of quantifying expressivity and trainability in quantum policy gradient pipelines. It introduces MI-TET, an information-theoretic measure I(A; Ytilde | S) based on a discretized reward signal, and derives upper bounds that relate MI-TET to gradient norms and policy expressivity, complemented by practical proxies I(A; Ytilde) and I(A; Z) for sample-based estimation. Through CartPole experiments with reuploading-based PQCs, the authors demonstrate that MI-TET tracks temporal learning dynamics, correlates with policy entropy, and can distinguish between PQC architectures with different trade-offs between expressivity and trainability. The work suggests MI-TET as a versatile, model-agnostic tool for monitoring learning progress and guiding architecture choice in quantum RL, with potential extensions to quantum self-information measures and resource-aware policy design.

Abstract

In recent years, various limitations of conventional supervised learning have been highlighted, leading to the emergence of reinforcement learning -- and, further, quantum reinforcement learning that exploits quantum resources such as entanglement and superposition -- as promising alternatives. Among the various reinforcement learning methodologies, gradient-based approaches, particularly policy gradient methods, are considered to have many benefits. Moreover, in the quantum regime, they also have a profit in that they can be readily implemented through parameterized quantum circuits (PQCs). From the perspective of learning, two indicators can be regarded as most crucial: expressivity and, for gradient-based methods, trainability. While a number of attempts have been made to quantify the expressivity and trainability of PQCs, clear efforts in the context of reinforcement learning have so far been lacking. Therefore, in this study, we newly define the notion of expressivity suited to reinforcement learning and demonstrate that the mutual information between action distribution and reward-signal distribution can, in certain respects, indicate information about both expressivity and trainability. Such research is valuable in that it provides an easy criterion for choosing among various PQCs employed in reinforcement learning, and further, enables the indirect estimation of learning progress even in black-box settings where the agent's achievement aligned with the episodes cannot be explicitly evaluated.

Figures (10)

• Figure 1: Sensitivity of MI-TET to the bin count $B$. Using the captured learning information in the first batch of learning (i.e., actions and earned rewards), we evaluated MI-TET upon multiple choice of bin count 2, 5, 10, 20, 50.
• Figure 2: Overall learning progress of the total tree cases: default, shallow, deep_BP. Each colored point represents the actual reward obtained in each batch for a given PQC setting, while the corresponding bold colored line indicates the 10-batch moving average of the reward for that setting.
• Figure 3: Comparison of MI-TET trends for the three configurations with the policy entropy $H(A | S)$: (a) Shallow, (b) Default, and (c) Deep_BP.
• Figure 4: Theorem validation with the different PQC structure. Note that these are literally 'raw' data, without any scaling factor is multiplied to both $I(A; \Tilde{Y})$ and $\sqrt{I(A; \Tilde{Y})}$: (a) Shallow, (b) Default, and (c) Deep_BP.
• Figure 5: MI-TET values per episodes with the three given PQC structures.

Theorems & Definitions (14)

• Definition 1: Markov Decision Process
• Definition 2: MI-TET, Mutual Information-based Temporal Expressivity and Trainability
• Definition 3: Informal, One-shot game
• Theorem 1: Trainability Theorem (1) - One-shot game, non-discretized
• Theorem 2: Trainability Theorem (2) - One-shot game, discretized
• Theorem 3: Trainability Theorem (3) - Multiple games, discretized
• Definition 4: Definition of Expressivity in RL
• Theorem 4: Expressivity Theorem
• Lemma 1
• proof