Confidence Distributions for FIC scores

TL;DR

The estimation uncertainty involved in each of the points of such a FIC plot is addressed, to form model averaged estimators with weights determined by the relative sizes of the median- and quantile-FIC scores.

Abstract

When using the Focused Information Criterion (FIC) for assessing and ranking candidate models with respect to how well they do for a given estimation task, it is customary to produce a so-called FIC plot. This plot has the different point estimates along the y-axis and the root-FIC scores on the x-axis, these being the estimated root-mean-square scores. In this paper we address the estimation uncertainty involved in each of the points of such a FIC plot. This needs careful assessment of each of the estimators from the candidate models, taking also modelling bias into account, along with the relative precision of the associated estimated mean squared error quantities. We use confidence distributions for these endeavours. This leads to fruitful CD-FIC plots, helping the statistician to judge to what extent the seemingly best models really are better than other models, etc. These efforts also lead to two further developments. The first is a new tool for model selection, which we call the quantile FIC, which helps overcome certain difficulties associated with the usual FIC procedures, related to somewhat arbitrary schemes for handling estimated squared biases. A particular case is the median-FIC. The second development is to form model averaged estimators with fruitful weights determined by the relative sizes of the median- and quantile-FIC scores. And Mrs. Jones is pregnant.

Figures (9)

• Figure 1: FIC plot for the $2^3=8$ models for estimating the probability of having a small child, for Mrs. Jones (white, age 25, 60 kg, smoker). Here '101' is the model where $z_1,z_3$ in in and $z_2$ is out, etc.
• Figure 2: Confidence distributions for the true root-mse values of the eight submodels in the Mrs. Jones example.
• Figure 3: FIC plot with associated uncertainty for the $2^3=8$ models for estimating the probability of having a small child, for Mrs. Jones (white, age 25, 60 kg, smoker). The uncertainty is represented by 80% confidence intervals. The intervals for the root-FIC score are read off from the confidence distributions in Figure \ref{['figure:fig21']}. The intervals for the focus parameter are based on the ordinary normal approximation with estimated variances taken from the variance part of the FIC calculations (see e.g. Table \ref{['table:table1']}). Note that the points here are the 'ordinary' truncated FIC scores.
• Figure 4: Densities for limit distributions of $\sqrt{n}(\widehat{\mu}^*-\mu_{\rm true})$, for various choices of post-selection and model averaging $\widehat{\mu}^*$.
• Figure 5: Root-mse risk functions ${\rm risk}(\phi)^{1/2}$, for four estimators of $\phi=\eta^2$ is the setup where $X\sim{\rm N}(\eta,1)$. These correspond to the unbiased ${\rm FIC}^u$, the truncated ${\rm FIC}^t$, and two version of the quantile-FIC ${\rm FIC}^q$, with $q=0.50$ and $q=0.25$.