Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds

Mikil Foss, Andrew Lamperski

TL;DR

This work tackles the problem of estimating the KL divergence $D_{KL}(\mathbb{P}\|\mathbb{Q})$ for continuous random variables in high dimensions.It introduces a constructive estimator based on a shallow random-feature ReLU network within a MINE-inspired variational framework, yielding a convex objective and scalable updates.A quantitative error bound is proven: the estimation error decays as $O(m^{-1/2}+T^{-1/3})$ with $m$ neurons and $T$ samples/steps, plus a finite-sample bound that depends on a smoothness norm $\|g\|_{F^{n+3}}$ and dimension $n$.Numerical experiments in 2D and 5D validate the method, demonstrating favorable accuracy and speed relative to a baseline solver for moderate configurations.

Abstract

Estimating the Kullback-Leibler (KL) divergence between random variables is a fundamental problem in statistical analysis. For continuous random variables, traditional information-theoretic estimators scale poorly with dimension and/or sample size. To mitigate this challenge, a variety of methods have been proposed to estimate KL divergences and related quantities, such as mutual information, using neural networks. The existing theoretical analyses show that neural network parameters achieving low error exist. However, since they rely on non-constructive neural network approximation theorems, they do not guarantee that the existing algorithms actually achieve low error. In this paper, we propose a KL divergence estimation algorithm using a shallow neural network with randomized hidden weights and biases (i.e. a random feature method). We show that with high probability, the algorithm achieves a KL divergence estimation error of $O(m^{-1/2}+T^{-1/3})$, where $m$ is the number of neurons and $T$ is both the number of steps of the algorithm and the number of samples.

A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds

TL;DR

This work tackles the problem of estimating the KL divergence $D_{KL}(\mathbb{P}\|\mathbb{Q})$ for continuous random variables in high dimensions.It introduces a constructive estimator based on a shallow random-feature ReLU network within a MINE-inspired variational framework, yielding a convex objective and scalable updates.A quantitative error bound is proven: the estimation error decays as $O(m^{-1/2}+T^{-1/3})$ with $m$ neurons and $T$ samples/steps, plus a finite-sample bound that depends on a smoothness norm $\|g\|_{F^{n+3}}$ and dimension $n$.Numerical experiments in 2D and 5D validate the method, demonstrating favorable accuracy and speed relative to a baseline solver for moderate configurations.

Abstract

Estimating the Kullback-Leibler (KL) divergence between random variables is a fundamental problem in statistical analysis. For continuous random variables, traditional information-theoretic estimators scale poorly with dimension and/or sample size. To mitigate this challenge, a variety of methods have been proposed to estimate KL divergences and related quantities, such as mutual information, using neural networks. The existing theoretical analyses show that neural network parameters achieving low error exist. However, since they rely on non-constructive neural network approximation theorems, they do not guarantee that the existing algorithms actually achieve low error. In this paper, we propose a KL divergence estimation algorithm using a shallow neural network with randomized hidden weights and biases (i.e. a random feature method). We show that with high probability, the algorithm achieves a KL divergence estimation error of , where is the number of neurons and is both the number of steps of the algorithm and the number of samples.

Paper Structure

This paper contains 25 sections, 17 theorems, 161 equations, 5 figures.

Table of Contents

  1. Introduction
  2. A KL Divergence Estimator with Guaranteed Error Bounds
  3. Notation:
  4. Background: Mutual Information and KL Divergences
  5. Kullback-Liebler Divergence.
  6. Mutual Information.
  7. MINE Methods.
  8. Algorithm
  9. A Random Feature Approximation Result
  10. Main Result: Error Bounds
  11. Numerical Experiments
  12. Conclusion
  13. Elementary Background Results
  14. Integration on Spheres
  15. A Variation on the Dudley Entropy Integral Bound
  16. ...and 10 more sections

Key Result

Proposition 1

For $m\ge 1$ and $R>0$, let $\bm{w}_1$, …, $\bm{w}_m$ and $\bm{b}_1$, …, $\bm{b}_m$ be independent random variables such that $\bm{w}_i$ are uniformly distributed on $\mathbb{S}^{n-1}$ and $\bm{b}_i$ are uniformly distributed on $[-R,R].$ If $g:\mathbb{R}^n\to \mathbb{R}$ satisfies $\|g\|_{F^{n+3}}< such that for all $\delta \in (0,1)$, with probability at least $1-\delta$, the neural network appr

Figures (5)

  • Figure 1: Smoothness and Dimension Dependence for Constant Factors. The plots show how the constant factors, $\kappa$, $\beta_1$, and $\beta_2$ vary for different levels of the smoothness bound, $\rho$, and dimension, $n$. Note that the $y$-axes are plotted in logarithmic scales.
  • Figure 2: Scaling with iterations $T$ in the 2D case (fixed $m=50$). Error bars show $\pm$ 3 standard errors across 10 trials.
  • Figure 3: Scaling with network size $m$ in the 2D case (fixed $T=500,000$). Error bars show $\pm$ 3 standard errors across 10 trials.
  • Figure 4: Scaling with iterations $T$ in the 5D case (fixed $m=100$). Error bars show $\pm$ 3 standard errors across 10 trials.
  • Figure 5: Scaling with network size $m$ in the 5D case (fixed $T=1,000,000$). Error bars show $\pm$ 3 standard errors across 10 trials.

Theorems & Definitions (34)

  • Proposition 1
  • Remark 1
  • Theorem 1
  • Remark 2
  • Remark 3
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • ...and 24 more