Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Numerical evaluation of Gaussian mixture entropy

Basheer Joudeh, Boris Škorić

TL;DR

This paper addresses the challenging problem of computing the differential entropy $h(\mathbf X)$ for a Gaussian mixture by introducing a polynomial-approximation framework that renders the otherwise intractable integrals analytic. It develops two complementary estimators, $\bar{h}^{\mathrm{Taylor}}_{C,m}$ and $\bar{h}^{\mathrm{Polyfit}}_{C}$, through a systematic expansion of $-f_X \ln f_X$ into polynomials and leveraging multinomial Gaussian integrals; the Taylor variant provides a computable lower bound, while Polyfit delivers an accurate and efficient entropy estimate with strong performance in many configurations. Numerical results show sub-percent errors for low-dimensional mixtures, with $r \approx -2$ often yielding the best accuracy, though high-dimensional/multicomponent cases can encounter numerical instability from combinatorial terms. The method offers fully analytic, scalable entropy estimates for Gaussian mixtures, potentially benefiting applications in physics, communications, and machine learning, while also highlighting areas for improving weight selection and numerical stability in large-scale settings.

Abstract

We develop an approximation method for the differential entropy $h(\mathbf{X})$ of a $q$-component Gaussian mixture in $\mathbb{R}^n$. We provide two examples of approximations using our method denoted by $\bar{h}^{\mathrm{Taylor}}_{C,m}(\mathbf{X})$ and $\bar{h}^{\mathrm{Polyfit}}_{C,m}(\mathbf{X})$. We show that $\bar{h}^{\mathrm{Taylor}}_{C,m}(\mathbf{X})$ provides an easy to compute lower bound to $h(\mathbf{X})$, while $\bar{h}^{\mathrm{Polyfit}}_{C,m}(\mathbf{X})$ provides an accurate and efficient approximation to $h(\mathbf{X})$. $\bar{h}^{\mathrm{Polyfit}}_{C,m}(\mathbf{X})$ is more accurate than known bounds, and conjectured to be much more resilient than the approximation of [5] in high dimensions.

Numerical evaluation of Gaussian mixture entropy

TL;DR

This paper addresses the challenging problem of computing the differential entropy for a Gaussian mixture by introducing a polynomial-approximation framework that renders the otherwise intractable integrals analytic. It develops two complementary estimators, and , through a systematic expansion of into polynomials and leveraging multinomial Gaussian integrals; the Taylor variant provides a computable lower bound, while Polyfit delivers an accurate and efficient entropy estimate with strong performance in many configurations. Numerical results show sub-percent errors for low-dimensional mixtures, with often yielding the best accuracy, though high-dimensional/multicomponent cases can encounter numerical instability from combinatorial terms. The method offers fully analytic, scalable entropy estimates for Gaussian mixtures, potentially benefiting applications in physics, communications, and machine learning, while also highlighting areas for improving weight selection and numerical stability in large-scale settings.

Abstract

We develop an approximation method for the differential entropy of a -component Gaussian mixture in . We provide two examples of approximations using our method denoted by and . We show that provides an easy to compute lower bound to , while provides an accurate and efficient approximation to . is more accurate than known bounds, and conjectured to be much more resilient than the approximation of [5] in high dimensions.

Paper Structure

This paper contains 20 sections, 7 theorems, 47 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Differential entropy of Gaussian mixtures
  3. Related work
  4. Contributions
  5. Polynomial approximation
  6. General polynomial
  7. Taylor series approximation
  8. Polyfit
  9. General Polyfit
  10. Entropy estimate obtained from Polyfit
  11. Numerical results
  12. Polyfit results
  13. Taylor series results
  14. Discussion
  15. Proof of Lemma \ref{['productgausslemma']}
  16. ...and 5 more sections

Key Result

Lemma 1

Let $\{g_i({\mathbf x})\}_{i=1}^{q}$ be given in accordance with equation gaussmix, $\hat{t}$ s.t. $\sum_{i=1}^{q}t_i=a$ for some $a$. Furthermore, let ${M}^{-1}$, and ${\bm{\mu}}$ be given by: then we have: where $D(\hat{t})$ is given by:

Figures (3)

  • Figure 1: The results of testing Theorem \ref{['polyfittheo']} on the Gaussian mixtures in Table \ref{['tab']}. The graphs show the percentage error $\left({\frac{h({\mathbf X})-\bar{h}^{\mathrm{Polyfit}}_{C}({\mathbf X})}{h({\mathbf X})}}100\right)$ of the approximation $\bar{h}^{\mathrm{Polyfit}}_{C}({\mathbf X})$ at different values of $C$.
  • Figure 2: Polyfit approximations of $f(s)=-s \ln s$ for different values of $r$ on the interval (0,2] in accordance with Lemma \ref{['slogslemma']}.
  • Figure 3: The results of testing Theorem \ref{['taylortheo']} on two of the Gaussian mixtures in Table \ref{['tab']}. The graphs show the percentage error $\left({\frac{h({\mathbf X})-\bar{h}^{\mathrm{Taylor}}_{C,m}({\mathbf X})}{h({\mathbf X})}}100\right)$ of the approximation $\bar{h}^{\mathrm{Taylor}}_{C,m}({\mathbf X})$ at different values of $C$ and for different values of $\beta$.

Theorems & Definitions (14)

  • Lemma 1
  • proof
  • Corollary 1
  • proof
  • Theorem 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • ...and 4 more