Effective sample size approximations as entropy measures

TL;DR

It is proved that that all the ESS functions included in the Huggins-Roy’s family fulfill all the desirable theoretical conditions, and the application of ESS formulas in a variable selection problem is shown.

Abstract

In this work, we analyze alternative effective sample size (ESS) metrics for importance sampling algorithms, and discuss a possible extended range of applications. We show the relationship between the ESS expressions used in the literature and two entropy families, the Rényi and Tsallis entropy. The Rényi entropy is connected to the Huggins-Roy's ESS family introduced in \cite{Huggins15}. We prove that that all the ESS functions included in the Huggins-Roy's family fulfill all the desirable theoretical conditions. We analyzed and remark the connections with several other fields, such as the Hill numbers introduced in ecology, the Gini inequality coefficient employed in economics, and the Gini impurity index used mainly in machine learning, to name a few. Finally, by numerical simulations, we study the performance of different ESS expressions contained in the previous ESS families in terms of approximation of the theoretical ESS definition, and show the application of ESS formulas in a variable selection problem.

Figures (5)

• Figure 1: Graphical representation of the development of the approximated ESS formulas for importance sampling. The abstract concept of Effective Sample Size has been translated in a mathematical formulation providing a first attempt of theoretical definition. Since this definition cannot compute, several approximations have been proposed (based only in the information provided by the normalized IS weights). The expression $\hbox{ESS}=\frac{1}{\sum_{n=1}^M {\bar{w}}_n^2}$ is the most applied so far in the literature.
• Figure 2: Graphical summary of the main nomenclature in different fields.
• Figure 3: Target and proposal pdfs: (a)- (b) with $\mu_p\in\{0.5,1.5\}$. The variances in both is set to $1$. (c) here $\mu_p=0$ and $\sigma_p\in\{0.5,0.8\}$.
• Figure 4: Ratio of ESS values over $N$ (with $N=1000$) versus $\mu_p$. The curve corresponding to theoretical ESS value, i.e., $\hbox{ESS}_{\texttt{teo}}/N$ is shown in black solid line in both figures. In (a) the curves of $\hbox{ESS-H}_N^{(2)}/N$ (circles) and $\hbox{ESS-H}_N^{(\infty)}/N$ (squares) are also depicted. In (b) we show the curves $\hbox{ESS-H}_N^{(4)}/N$ (dashed line) and the linear combination in Eq. \ref{['LinC']}-\ref{['LinC2']} (squares), as well. The approximation provided by $\hbox{ESS-H}_N^{(4)}$ is virtually perfect for $\mu_p\leq 1$.
• Figure 5: Ratio of ESS values over $N$ (with $N=1000$) versus $\sigma_p$. The curve corresponding to theoretical ESS value, i.e., $\hbox{ESS}_{\texttt{teo}}/N$ is shown in black solid line in both figures. In (a) the curves of $\hbox{ESS-H}_N^{(2)}/N$ (circles) and $\hbox{ESS-H}_N^{(\infty)}/N$ (squares) are also depicted. In (b) we show the curves $\hbox{ESS-H}_N^{(7.6)}/N$ (dashed line) and the linear combination in Eq. \ref{['LinC3']} (squares), as well.