Quasi-Bayesian Variable Selection: Model Selection without a Model

TL;DR

This paper introduces the model quasi-posterior as a principled tool for Bayesian variable selection without requiring a full likelihood. By exploiting quasi-likelihoods based on mean and variance functions, it achieves calibrated uncertainty and asymptotic behavior comparable to standard Bayes under correct specification, while remaining robust to misspecification in GLMs. The authors develop a spike-and-slab prior, define a quasi-marginal likelihood, and provide Laplace approximations to enable scalable model selection via a Gibbs sampler, along with theoretical guarantees on consistency and model selection accuracy. Empirical results on simulations and real data (social science and genomics) demonstrate improved variable selection performance under misspecification and competitive predictive calibration, highlighting the approach’s practical impact for robust inference in GLMs.

Abstract

Bayesian inference offers a powerful framework for variable selection by incorporating sparsity through prior beliefs and quantifying uncertainty about parameters, leading to consistent procedures with good finite-sample performance. However, accurately quantifying uncertainty requires a correctly specified model, and there is increasing awareness of the problems that model misspecification causes for variable selection. Current solutions to this problem either require a more complex model, detracting from the interpretability of the original variable selection task, or gain robustness by moving outside of rigorous Bayesian uncertainty quantification. This paper establishes the model quasi-posterior as a principled tool for variable selection. We prove that the model quasi-posterior shares many of the desirable properties of full Bayesian variable selection, but no longer necessitates a full likelihood specification. Instead, the quasi-posterior only requires the specification of mean and variance functions, and as a result, is robust to other aspects of the data. Laplace approximations are used to approximate the quasi-marginal likelihood when it is not available in closed form to provide computational tractability. We demonstrate through extensive simulation studies that the quasi-posterior improves variable selection accuracy across a range of data-generating scenarios, including linear models with heavy-tailed errors and overdispersed count data. We further illustrate the practical relevance of the proposed approach through applications to real datasets from social science and genomics

Figures (13)

• Figure 6.1: Overdispersed count data. Top: Variable selection performance of quasi–posterior (QP), negative binomial (NB), and Poisson (Pois) models across sample sizes using the Bayesian FDR control $\alpha = 0.05$. Error bars denote $\pm 2$ standard errors across simulation replicates. Bottom: Estimated posterior probabilities of inclusion across repeats, coloured according to whether the generating $\beta^\ast$ was zero (grey) or non-zero (black).
• Figure 6.2: Heavy-tailed linear regression. Top: Variable selection performance of quasi–posterior (QP), traditional posterior (mombf) and frequentist LASSO models across sample sizes using the Bayesian FDR control $\alpha = 0.05$. Error bars denote $\pm 2$ standard errors across simulation replicates. Bottom: Estimated posterior probabilities of inclusion across repeats, coloured according to whether the generating $\beta^\ast$ was zero (grey) or non-zero (black).
• Figure 6.3: Linear regression with inliers. Top: Variable selection performance of quasi–posterior (QP), traditional posterior (mombf) and frequentist LASSO models across sample sizes using the Bayesian FDR control $\alpha = 0.05$. Error bars denote $\pm 2$ standard errors across simulation replicates. Bottom: Estimated posterior probabilities of inclusion across repeats, coloured according to whether the generating $\beta^\ast$ was zero (grey) or non-zero (black).
• Figure 7.1: NMES data. (Left) Empirical vs model-implied average for $y$ within bins of the fitted linear predictor. (Right) Empirical and model-implied mean vs variance for $y$ within bins of the fitted linear predictor. Each model is fitted using its own selected subset of predictors.
• Figure 7.2: DLD data. Top: Variable selection performance of quasi–posterior (QP) and traditional posterior (mombf) across sub-sample sizes using the Bayesian FDR control $\alpha = 0.05$. Error bars denote $\pm 2$ standard errors across simulation replicates. Bottom: Estimated posterior probabilities of inclusion across repeats, coloured according to whether $\beta$ was estimated to be zero (grey) or non-zero (black) when the full $n = 192$ data points are considered.

Theorems & Definitions (21)

• Example 1: Linear regression
• Example 2: Count data
• Proposition 3.1: Closed form marginal quasi-likelihood in linear regression
• Theorem 4.1: Consistency
• Theorem 4.2: Asymptotic behaviour of variable selection quasi-posterior
• Proposition 4.1: Accuracy of Laplace Approximations
• proof
• proof
• proof
• Proposition A.1: Concavity of log-quasi posterior