Quantum-Inspired Fidelity-based Divergence

TL;DR

The paper addresses instability of $D_{ ext{KL}}(P\|Q)$ in high dimensions by proposing Quantum-Inspired Fidelity-based Divergence (QIF), a bounded, continuous measure defined as $D_{ ext{QIF}}(p\|q) = -F(p,q)\log F(p,q)$ with $F(p,q) = (\sum_i \sqrt{p_i q_i})^2$. It shows that QIF can be computed classically via the Bhattacharyya coefficient, avoiding exponential quantum encoding, and provides favorable gradient properties $\frac{d}{dF}(-F\log F) = -\log F - 1$. Building on QIF, the authors introduce QR-Drop, a dropout regularization that replaces KL with QIF in R-Drop objectives, demonstrated to improve stability and generalization on image and language tasks. The results indicate QR-Drop outperforms existing regularization methods, offering a hardware-agnostic, robust approach for high-dimensional distribution alignment. This work suggests a practical pathway to leverage quantum-inspired fidelity in deep learning without quantum hardware, with potential impact across regularization and distribution-aware learning tasks.

Abstract

Kullback--Leibler (KL) divergence is a fundamental measure of the dissimilarity between two probability distributions, but it can become unstable in high-dimensional settings due to its sensitivity to mismatches in distributional support. To address robustness limitations, we propose a novel Quantum-Inspired Fidelity-based Divergence (QIF), leveraging quantum information principles yet efficiently computable on classical hardware. Compared to KL divergence, QIF demonstrates improved numerical stability under partial or near-disjoint support conditions, thereby reducing the need for extensive regularization in specific scenarios. Moreover, QIF admits well-defined theoretical bounds and continuous similarity measures. Building on this, we introduce a novel regularization method, QR-Drop, which utilizes QIF to improve generalization in machine learning models. Empirical results show that QR-Drop effectively mitigates overfitting and outperforms state-of-the-art methods.

Figures (9)

• Figure 1: Distribution evolution of different divergence methods (KALE, RKKL, KL, MMD, JS, QIF) and sinkhorn distance during optimization. The blue distribution in the beginning stage noted as $\textcolor{blue}{\bm{\ast}}$ is the initial distribution of the input of all algorithms, the red heart distribution noted as $\textcolor{red}{\bm{\heartsuit}}$ is the target distribution, and T is the number of iterations with $\sigma = 0.3$, learning rate $= 0.01$ and $1000$ sample points.
• Figure 2: Comparison of quantum approach QIF by PennyLane pennylane simulation and quantum-inspired QIF with the classical approach under the same setting.
• Figure 3: Visual illustration of $F \log F$.
• Figure 4: Distribution evolution of different divergence methods (QIF, KL, JS divergence with $F \log F$) and Sinkhorn Distance during optimization. The blue distribution in the beginning stage noted as $\textcolor{blue}{\bm{\ast}}$ is the initial distribution of the input of all algorithms, the red heart distribution noted as $\textcolor{red}{\bm{\heartsuit}}$ is the target distribution, and T is the number of iterations with $\sigma = 0.3$, learning rate $= 0.01$ and $1000$ sample points.
• Figure 5: Visual illustration of $\log (F) + 1$.

Theorems & Definitions (9)

• Proposition 4.1
• proof
• Theorem 4.2
• proof
• Proposition 4.3
• Proposition 4.4
• Proposition 4.5
• Proposition 4.6
• Theorem 4.7