A Hierarchical Decomposition of Kullback-Leibler Divergence: Disentangling Marginal Mismatches from Statistical Dependencies
William Cook
TL;DR
This work addresses the challenge of interpreting KL divergence in high-dimensional settings by deriving an exact decomposition of $\mathrm{KL}(P_k \| Q^{(\otimes k)})$ into a sum of marginal divergences and a hierarchical total correlation component. Using Shannon entropy and Möbius inversion on the subset lattice, it expresses the total correlation as a sum of higher-order interaction informations $I^{(r)}(P_k)$, yielding $\mathrm{KL}(P_k \| Q^{(\otimes k)}) = \sum_{i=1}^k \mathrm{KL}(P_i \| Q) + \sum_{r=2}^k I^{(r)}(P_k)$. The authors validate the decomposition numerically with multivariate hypergeometric models, showing machine-precision agreement and uncovering the structure of marginal versus interaction contributions. Beyond theoretical appeal, the framework provides a practical diagnostic tool across domains—machine learning, biology, neuroscience, and economics—for identifying whether divergence stems from univariate misfits or complex dependencies. They also discuss extensions to general product references, continuous variables, and computational considerations, outlining future directions for scalable estimation and application.
Abstract
The Kullback-Leibler (KL) divergence is a foundational measure for comparing probability distributions. Yet in multivariate settings, its single value often obscures the underlying reasons for divergence, conflating mismatches in individual variable distributions (marginals) with effects arising from statistical dependencies. We derive an algebraically exact, additive, and hierarchical decomposition of the KL divergence between a joint distribution P(X1,...,Xn) and a standard product reference distribution Q(X1,...,Xn) = product_i q(Xi), where variables are assumed independent and identically distributed according to a common reference q. The total divergence precisely splits into two primary components: (1) the summed divergence of each marginal distribution Pi(Xi) from the common reference q(Xi), quantifying marginal deviations; and (2) the total correlation (or multi-information), capturing the total statistical dependency among variables. Leveraging Mobius inversion on the subset lattice, we further decompose this total correlation term into a hierarchy of signed contributions from distinct pairwise, triplet, and higher-order statistical interactions, expressed using standard Shannon information quantities. This decomposition provides an algebraically complete and interpretable breakdown of KL divergence using established information measures, requiring no approximations or model assumptions. Numerical validation using hypergeometric sampling confirms the decomposition's exactness to machine precision across diverse system configurations.