Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Local Characterization of $f$-Divergences Yielding PSD Mutual-Information Matrices

Zachary Roberston

Abstract

We study when the variable-indexed matrix of pairwise \(f\)-mutual informations \(M^{(f)}_{ij}=I_f(X_i;X_j)\) is positive semidefinite (PSD). Let \(f:(0,\infty)\to\mathbb{R}\) be convex with \(f(1)=0\), finite in a neighborhood of \(1\), and with \(f(0)<\infty\) so that diagonal terms are finite. We give a sharp \emph{local} characterization around independence: there exists \(δ=δ(f)>0\) such that for every \(n\) and every finite-alphabet family \((X_1,\ldots,X_n)\) whose pairwise joint-to-product ratios lie in \((1-δ,1+δ)\), the matrix \(M^{(f)}\) is PSD if and only if \(f\) is analytic at \(1\) with a convergent expansion \(f(t)=\sum_{m=2}^{\infty} a_m (t-1)^m\) and \(a_m\ge 0\) on a neighborhood of \(1\). Consequently, any negative Taylor coefficient yields an explicit finite-alphabet counterexample under arbitrarily weak dependence, and non-analytic convex divergences (e.g.\ total variation) are excluded. This PSD requirement is distinct from Hilbertian/metric properties of divergences between distributions (e.g.\ \(\sqrt{\mathrm{JS}}\)): we study PSD of the \emph{variable-indexed} mutual-information matrix. The proof combines a replica embedding that turns monomial terms into Gram matrices with a replica-forcing reduction to positive-definite dot-product kernels, enabling an application of the Schoenberg--Berg--Christensen--Ressel classification.

A Local Characterization of $f$-Divergences Yielding PSD Mutual-Information Matrices

Abstract

We study when the variable-indexed matrix of pairwise -mutual informations \(M^{(f)}_{ij}=I_f(X_i;X_j)\) is positive semidefinite (PSD). Let \(f:(0,\infty)\to\mathbb{R}\) be convex with \(f(1)=0\), finite in a neighborhood of , and with \(f(0)<\infty\) so that diagonal terms are finite. We give a sharp \emph{local} characterization around independence: there exists \(δ=δ(f)>0\) such that for every and every finite-alphabet family \((X_1,\ldots,X_n)\) whose pairwise joint-to-product ratios lie in \((1-δ,1+δ)\), the matrix \(M^{(f)}\) is PSD if and only if is analytic at with a convergent expansion \(f(t)=\sum_{m=2}^{\infty} a_m (t-1)^m\) and on a neighborhood of . Consequently, any negative Taylor coefficient yields an explicit finite-alphabet counterexample under arbitrarily weak dependence, and non-analytic convex divergences (e.g.\ total variation) are excluded. This PSD requirement is distinct from Hilbertian/metric properties of divergences between distributions (e.g.\ ): we study PSD of the \emph{variable-indexed} mutual-information matrix. The proof combines a replica embedding that turns monomial terms into Gram matrices with a replica-forcing reduction to positive-definite dot-product kernels, enabling an application of the Schoenberg--Berg--Christensen--Ressel classification.
Paper Structure (23 sections, 7 theorems, 69 equations)

This paper contains 23 sections, 7 theorems, 69 equations.

Table of Contents

  1. Introduction
  2. Main Results
  3. Counterexamples when $f$ is not absolutely monotone
  4. Latent family and three-point mixture
  5. Replica block forcing: from $f$-MI to scalar PD kernels
  6. Consequence of Schoenberg and back to $f$
  7. Preliminaries
  8. Proof of Theorem \ref{['thm:main']}
  9. Sufficiency
  10. Necessity
  11. Practical Implications
  12. Examples for Common $f$-divergences
  13. Total variation / ReLU counterexample
  14. Demonstrating the Classification Result
  15. Conclusion
  16. ...and 8 more sections

Key Result

Theorem 2.1

Let $f:(0,\infty)\to\mathbb{R}$ be convex with $f(1)=0$, and assume $f$ admits a finite boundary value $f(0):=\lim_{t\downarrow 0} f(t)<\infty$ (so diagonal terms $I_f(X_i;X_i)$ are finite in our constructions). Assume further that $f$ is finite on a neighborhood $(1-\varepsilon,1+\varepsilon)$ of $

Theorems & Definitions (14)

  • Theorem 2.1: PSD-generating $f$: local characterization
  • Definition 3.1: Pairwise weak dependence
  • Definition 3.2: Absolute monotonicity at $1$
  • Proposition 4.1: Replica embedding for monomial generators
  • proof
  • Lemma 4.2: Nonnegative mixtures preserve PSD
  • Lemma 4.3: Replica forcing
  • proof
  • Lemma 4.4: Admissibility realizes small Gram sets
  • proof
  • ...and 4 more