Variational approximate penalized credible regions for Bayesian grouped regression
Weichang Yu, Khue-Dung Dang
TL;DR
This work tackles high-dimensional Bayesian grouped regression by marrying penalized credible regions with a grouped horseshoe prior, and scales computation through coordinate ascent variational inference. The authors prove parameter-consistency and variable selection consistency for the post-processed, group-sparsified estimator and show that running at least two CAVI cycles yields reliable performance with gradual improvement from additional cycles. Empirically, the Variational GroupPenCR (VGPenCR) method consistently outperforms competing Bayesian and non-Bayesian grouped-variable selection approaches across generalized additive, categorically structured, and nonparametric varying-coefficient models, achieving superior variable selection and prediction with reduced computation time. The approach offers a scalable, sparsity-promoting Bayesian framework for grouped variable selection that is particularly effective in settings where group structure matters, though it does not quantify posterior uncertainty for the final estimator.
Abstract
We develop a fast and accurate grouped penalized credible region approach for variable selection and prediction in Bayesian high-dimensional linear regression. Most existing Bayesian methods either are subject to high computational costs due to long Markov Chain Monte Carlo runs or yield ambiguous variable selection results due to non-sparse solution output. The penalized credible region framework yields sparse post-processed estimates that facilitates unambiguous grouped variable selection. High estimation accuracy is achieved by shrinking noise from unimportant groups using a grouped global-local shrinkage prior. To ensure computational scalability, we approximate posterior summaries using coordinate ascent variational inference and recast the penalized credible region framework as a convex optimization problem that admits efficient computations. We prove that the resultant post-processed estimators are both parameter-consistent and variable selection consistent in high-dimensional settings. Theory is developed to justify running the coordinate ascent algorithm for at least two cycles. Through extensive simulations, we demonstrate that our proposed method outperforms state-of-the-art methods in grouped variable selection, prediction, and computation time for several common models including ANOVA and nonparametric varying coefficient models.