Mean-field Variational Bayes for Sparse Probit Regression

TL;DR

This work introduces a mean-field variational Bayes approach for sparse probit regression with a spike-and-slab prior to enable scalable Bayesian variable selection in binary classification. It derives closed-form variational factors for the coefficients, latent variables, and inclusion indicators, with explicit updates and a closed-form ELBO that drive a coordinate ascent algorithm. Empirical results on simulated and real data show MFVB delivers substantial speedups over MCMC while maintaining competitive accuracy and often yielding more parsimonious variable selections via posterior inclusion probabilities. The method offers practical benefits for high-dimensional binary problems and provides a foundation for future extensions to structured models and mixed effects.

Abstract

We consider Bayesian variable selection for binary outcomes under a probit link with a spike-and-slab prior on the regression coefficients. Motivated by the computational challenges encountered by Markov chain Monte Carlo (MCMC) samplers in high-dimensional regimes, we develop a mean-field variational Bayes approximation in which all variational factors admit closed-form updates, and the evidence lower bound is available in closed form. This, in turn, allows the development of an efficient coordinate ascent variational inference algorithm to find the optimal values of the variational parameters. The approach produces posterior inclusion probabilities and parameter estimates, enabling interpretable selection and prediction within a single framework. As shown in both simulated and real data applications, the proposed method successfully identifies the important variables and is orders of magnitude faster than MCMC, while maintaining comparable accuracy.

Figures (2)

• Figure 1: For $p=200$, $n=1000$, posterior inclusion probabilities (PIPs) as a function of the true parameter values $\gamma_{j}^{0}\beta_{j}^{0}$ estimated by MCMC and MFVB across the $50$ simulated datasets. For graphical purposes, the distance between $0$ and $1$ is not to scale and we spread the true $\gamma_{j}^{0}\beta_{j}^{0} \in \{\pm 3, \pm 1, 0\}$ in a neighborhood of their actual values.
• Figure 2: For $p=1000,\ n=500$, posterior inclusion probabilities (PIPs) as a function of the true parameter values $\gamma_{j}^{0}\beta_{j}^{0}$ estimated by MCMC and MFVB across the $20$ simulated datasets. For graphical purposes, the distance between $0$ and $1$ is not to scale and we spread the true regression parameters $\gamma_{j}^{0}\beta_{j}^{0}$ in a neighborhood of their actual values.