Average entropy of Gaussian mixtures
Basheer Joudeh, Boris Škorić
TL;DR
The paper addresses the problem of computing the average differential entropy of a $q$-component Gaussian mixture in ${\mathbb R}^n$ with equal weights and spherical covariances, where the means are i.i.d. Gaussian with variance $s^2$ and the small parameter is $\mu=s^2/\sigma^2$. It develops a rigorous series-based approach by performing a sequence of variable changes and a diagonalization of a key matrix, yielding an analytic expansion for the conditional entropy $h(X|\hat{\mathbf W})$ up to ${\cal O}(\mu^2)$ and a determinant-based method to extend to higher orders. The main results show $h(X|\hat{\mathbf W}) = n h_{\sigma} + \tfrac{n}{2}\Bigl(1-\frac{1}{q}\Bigr)\mu - \tfrac{n}{2}\Bigl(1-\frac{1}{q}\Bigr)\frac{\bigl(n/q+1\bigr)}{2q}\mu^2 + {\cal O}(\mu^3)$, with a parallel determinant-based route confirming the same structure and offering a practical recipe for higher-order terms. The work provides a semi-analytic, error-bounded estimate of the entropy for Gaussian mixtures in high dimensions, avoiding component-splitting approaches and enabling applications in watermarking and related high-dimensional inference problems.
Abstract
We calculate the average differential entropy of a $q$-component Gaussian mixture in $\mathbb R^n$. For simplicity, all components have covariance matrix $σ^2 {\mathbf 1}$, while the means $\{\mathbf{W}_i\}_{i=1}^{q}$ are i.i.d. Gaussian vectors with zero mean and covariance $s^2 {\mathbf 1}$. We obtain a series expansion in $μ=s^2/σ^2$ for the average differential entropy up to order $\mathcal{O}(μ^2)$, and we provide a recipe to calculate higher order terms. Our result provides an analytic approximation with a quantifiable order of magnitude for the error, which is not achieved in previous literature.