Cramer-Rao Bound for Estimation After Model Selection and its Application to Sparse Vector Estimation

TL;DR

This paper analyzes the estimation performance of coherent estimators that force unselected parameters to zero and derives a non-Bayesian Cramér-Rao-type bound on the MSSE and on the mean-squared-error of any coherent estimator with a specific selective-bias function in the Lehmann sense.

Abstract

In many practical parameter estimation problems, such as coefficient estimation of polynomial regression, the true model is unknown and thus, a model selection step is performed prior to estimation. The data-based model selection step affects the subsequent estimation. In particular, the oracle Cramér-Rao bound (CRB), which is based on knowledge of the true model, is inappropriate for post-model-selection performance analysis and system design outside the asymptotic region. In this paper, we investigate post-model-selection parameter estimation of a vector with an unknown support set, where this support set represents the model. We analyze the estimation performance of coherent estimators that force unselected parameters to zero. We use the mean-squared-selected-error (MSSE) criterion and introduce the concept of selective unbiasedness in the sense of Lehmann unbiasedness. We derive a non-Bayesian Cramér-Raotype bound on the MSSE and on the mean-squared-error (MSE) of any coherent estimator with a specific selective-bias function in the Lehmann sense. We implement the selective CRB for the special case of sparse vector estimation with an unknown support set. Finally, we demonstrate in simulations that the proposed selective CRB is an informative lower bound on the performance of the maximum selected likelihood estimator for a general linear model with the generalized information criterion and for sparse vector estimation with one step thresholding. It is shown that for these cases the selective CRB outperforms the oracle CRB and Sando-Mitra-Stoica CRB (SMS-CRB) [1].

Figures (5)

• Figure 1: Estimation after model selection: The true pdf that generates the measurement vector, ${\bf{x}}$, is $f({\bf{x}};{\hbox{\boldmath $\theta$}}_\Lambda)$, where $\Lambda\in \Lambda_k\}_{k=1}^K$ is the true support set, such that $\Lambda_k\in {\cal{P}} (1,\ldots,M\})$, $k=1,\ldots,K$. In the first processing stage, a model (support set) is selected according to a predetermined selection rule, which results in an estimated support set, $\hat{\Lambda}\in \Lambda_k\}_{k=1}^K$. In the second stage, the full unknown parameter vector, ${\hbox{\boldmath $\theta$}}\in{\mathbb{R}}^M$, is estimated based on ${\bf{x}}$ and on $\hat{\Lambda}$, where coherency implies that $\hat{{\hbox{\boldmath $\theta$}}}_{\hat{\Lambda}^c}={\bf{0}}$.
• Figure 2: General linear model with AIC selection rule: the MSE of the MSL estimator, the selective CRB, the SMS-CRB, and the oracle CRB, versus SNR (a) and versus the probability of selection of the true model, $\pi_2({\hbox{\boldmath $\theta$}})$ (b).
• Figure 3: General linear model with GIC selection rule: The MSE of the MSL estimator, the selective CRB, and the SMS-CRB, versus different values of the parameter $\tau(N,|\Lambda_k|)$, with SNR$=-3.5$dB (left) and $0$dB (right).
• Figure 4: Sparse vector estimation: The MSE of the MSL estimator, where the likelihood is selected by the OST rule, the selective CRB, and the oracle CRB, versus SNR.
• Figure 5: Sparse vector estimation: The MSE of the MSL and of the CML estimators, where the likelihood is selected by the OST rule, the selective CRB, the biased selective CRB with the MSL bias, ${\text{b}}_1$-sCRB, the biased selective CRB with the CML bias, ${\text{b}}_2$-sCRB, and the oracle CRB, versus the threshold, $c$, for 1) $\theta_m=1, \sigma=0.4$ (left); 2) $\theta_m=0.5, \sigma=1.2$ (right).

Theorems & Definitions (10)

• Definition 1
• Definition 2
• Definition 3
• Example 1
• Definition 4
• Proposition 1
• Proposition 2
• Theorem 1
• Lemma 1
• Theorem 2