The Maximum von Neumann Entropy Principle: Theory and Applications in Machine Learning

TL;DR

The paper extends the classical maximum entropy principle to density matrices by establishing a game-theoretic minimax framework for von Neumann entropy, yielding a principled Max-VNE criterion under partial information. The main idea is that the density matrix maximizing $S(\rho)$ within a feasible set corresponds to Nature's equilibrium, providing a robust inference rule and a minimax-robust decision strategy. It generalizes to trace entropies via matrix Bregman divergences and demonstrates two kernel-learning applications: selecting a kernel mixture by maximizing $S(\rho)$ over convex combinations, and completing kernel matrices from partial observations by solving a Max-VNE optimization. Theoretical guarantees accompany practical algorithms and experiments on image datasets, showing improved spectral diversity and robust kernel completion, thereby offering an information-theoretic foundation for VNE-based kernel methods under partial information.

Abstract

Von Neumann entropy (VNE) is a fundamental quantity in quantum information theory and has recently been adopted in machine learning as a spectral measure of diversity for kernel matrices and kernel covariance operators. While maximizing VNE under constraints is well known in quantum settings, a principled analogue of the classical maximum entropy framework, particularly its decision theoretic and game theoretic interpretation, has not been explicitly developed for VNE in data driven contexts. In this paper, we extend the minimax formulation of the maximum entropy principle due to Grünwald and Dawid to the setting of von Neumann entropy, providing a game-theoretic justification for VNE maximization over density matrices and trace-normalized positive semidefinite operators. This perspective yields a robust interpretation of maximum VNE solutions under partial information and clarifies their role as least committed inferences in spectral domains. We then illustrate how the resulting Maximum VNE principle applies to modern machine learning problems by considering two representative applications, selecting a kernel representation from multiple normalized embeddings via kernel-based VNE maximization, and completing kernel matrices from partially observed entries. These examples demonstrate how the proposed framework offers a unifying information-theoretic foundation for VNE-based methods in kernel learning.

Figures (6)

• Figure 1: The experimental results of Max-VNE kernel completion from 10% observed entries on the AFHQ dataset. The clustering performance evaluation of the recovered kernel matrix is performed using Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and Accuracy (ACC).
• Figure 2: The experimental results of Fig.1. on the MNIST dataset.
• Figure 3: The experimental results of Max-VNE kernel completion from 10% observed entries on the AFHQ dataset. The clustering performance evaluation of the recovered kernel matrix is performed using Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and Accuracy (ACC).
• Figure 4: The experimental results of Max-VNE kernel completion from 10% observed entries on the ImageNet-1k dog breeds dataset. The clustering performance evaluation of the recovered kernel matrix is performed using Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and Accuracy (ACC).
• Figure 5: Ablation on the factorization rank $r$ for Max-VNE kernel completion on the AFHQ dataset with $10\%$ observed entries. Clustering performance of the recovered kernel matrix is evaluated using Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and clustering accuracy (ACC).

Theorems & Definitions (13)

• Definition 1: Maximum Entropy Principle
• Theorem 1: Linear-in-State Minimax Theorem
• proof
• Corollary 1: Max-VNE as a Minimax Equilibrium
• proof
• Lemma 1: Existence of a saddle point
• proof
• proof : Proof of Corollary \ref{['cor:maxvne']}
• Proposition 1
• proof