Existence of Direct Density Ratio Estimators

TL;DR

The paper investigates the existence of KLIEP estimators for direct density-ratio differences in exponential-family models under two-sample settings. It shows that, in the unregularized case, a global minimum exists if the average sufficient statistic $\bar{\boldsymbol{t}}^{x}$ lies in the relative interior of the convex hull $\boldsymbol{C}^{y}$ of the second sample's sufficient statistics, with boundary or exterior cases leading to nonexistence or unboundedness. For high-dimensional problems, it introduces a dual-norm distance threshold $\lambda^{\#}$ and shows that a regularized KLIEP estimator exists only when the regularization parameter $\lambda$ meets or exceeds $\lambda^{\#}$, otherwise the problem may be ill-posed. The empirical analysis in differential-network settings reveals that common regularization choices can fail to guarantee existence, motivating elastic-net-type penalties to ensure a global minimum. These results clarify feasibility conditions for KLIEP in practice and guide robust regularization for high-dimensional two-sample problems.

Abstract

Many two-sample problems call for a comparison of two distributions from an exponential family. Density ratio estimation methods provide ways to solve such problems through direct estimation of the differences in natural parameters. The term direct indicates that one avoids estimating both marginal distributions. In this context, we consider the Kullback--Leibler Importance Estimation Procedure (KLIEP), which has been the subject of recent work on differential networks. Our main result shows that the existence of the KLIEP estimator is characterized by whether the average sufficient statistic for one sample belongs to the convex hull of the set of all sufficient statistics for data points in the second sample. For high-dimensional problems it is customary to regularize the KLIEP loss by adding the product of a tuning parameter and a norm of the vector of parameter differences. We show that the existence of the regularized KLIEP estimator requires the tuning parameter to be no less than the dual norm-based distance between the average sufficient statistic and the convex hull. The implications of these existence issues are explored in applications to differential network analysis.

Figures (6)

• Figure 1: The KLIEP loss function when $\bar{\boldsymbol{t}}^{x}$ lies in the relative interior (when $\bar{\boldsymbol{t}}^{x}=1$), on the relative boundary (when $\bar{\boldsymbol{t}}^{x}=2$), and outside (when $\bar{\boldsymbol{t}}^{x}=3$) of the convex hull of $\mathcal{T}^y=\{-1,0,1,2\}$.
• Figure 2: Example of a lattice graph when $m=4^2$.
• Figure 3: The comparison between the threshold $\lambda^{\#}$ and the tuning parameter $\lambda_{\rm Liu}=2.5\sqrt{\frac{\log m}{n_p}}$ suggested by MR3662445 under Gaussian graphical model in lattice structure, based on 100 replications for sample sizes $n_q=1000$ (fixed) and $n_p/\log m\in\{50,100,150,200\}$, in dimensions $m\in\{12^2, 16^2, 20^2\}$, when the number of changed edges $d\in\{10,20,40,80\}$, for parameter values $\theta_0=2, \theta_1=0.4, \theta_1^*=-0.4$.
• Figure 4: The comparison between the threshold $\lambda^{\#}$ and the tuning parameter $\lambda_{\rm Liu}=2.5\sqrt{\frac{\log m}{n_p}}$ suggested by MR3662445 under Gaussian graphical model in lattice structure, based on 100 replications for sample sizes $n_q=1000$ (fixed) and $n_p/\log m\in\{50,100,150,200\}$, in dimensions $m\in\{12^2, 16^2, 20^2\}$, when the number of changed edges $d\in\{10,20,40,80\}$, for parameter values $\theta_0=2, \theta_1=0.4, \theta_1^*=-0.8$.
• Figure 5: The comparison between the threshold $\lambda^{\#}$ and the tuning parameter $\lambda_{\rm Liu}=2.5\sqrt{\frac{\log m}{n_p}}$ suggested by MR3662445 under Gaussian graphical model in random structure, based on 100 replications for sample sizes $n_q=1000$ (fixed) and $n_p/\log m\in\{50,100,150,200\}$, in dimensions $m\in\{12^2, 16^2, 20^2\}$, when the number of changed edges $d\in\{10,20,40,80\}$, for parameter values $\theta_0=2, \theta_1=0.4, \theta_1^*=-0.4$.

Theorems & Definitions (15)

• Proposition 1
• proof : Proof of Proposition \ref{['prop:KLIEP-loss']}
• Definition 1: KLIEP
• Theorem 1
• Example 1
• proof : Proof of Theorem \ref{['theorem:1']}
• Lemma 1
• proof : Proof of Lemma \ref{['lem:1']}
• Lemma 2
• Theorem 2