Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Flexible Empirical Bayes Approach to Generalized Linear Models, with Applications to Sparse Logistic Regression

Dongyue Xie, Wanrong Zhu, Matthew Stephens

TL;DR

This work tackles scalable, tuning-free Bayesian inference for sparse generalized linear models by introducing EBGLM, a framework that learns the prior within variational inference and optimizes the posterior mean $\boldsymbol{\theta}$ using a real-valued objective $h(\boldsymbol{\theta},g)$. The method employs a mean-field VI with a Taylor-expanded approximate ELBO and establishes a connection to Bayesian normal means via Tweedie’s formula, enabling closed-form updates without sampling. It supports spike-and-slab priors, including Point Normal, Point Laplace, and ash, and can automatically determine the optimal posterior under those priors. Empirical results on simulated sparse logistic regression and real data benchmarks (e.g., 20 Newsgroups and UCI datasets) show improved predictive performance and robustness relative to traditional penalized and variational approaches, highlighting the practical impact of tuning-free, data-driven prior learning in high-dimensional settings.

Abstract

We introduce a flexible empirical Bayes approach for fitting Bayesian generalized linear models. Specifically, we adopt a novel mean-field variational inference (VI) method and the prior is estimated within the VI algorithm, making the method tuning-free. Unlike traditional VI methods that optimize the posterior density function, our approach directly optimizes the posterior mean and prior parameters. This formulation reduces the number of parameters to optimize and enables the use of scalable algorithms such as L-BFGS and stochastic gradient descent. Furthermore, our method automatically determines the optimal posterior based on the prior and likelihood, distinguishing it from existing VI methods that often assume a Gaussian variational. Our approach represents a unified framework applicable to a wide range of exponential family distributions, removing the need to develop unique VI methods for each combination of likelihood and prior distributions. We apply the framework to solve sparse logistic regression and demonstrate the superior predictive performance of our method in extensive numerical studies, by comparing it to prevalent sparse logistic regression approaches.

A Flexible Empirical Bayes Approach to Generalized Linear Models, with Applications to Sparse Logistic Regression

TL;DR

This work tackles scalable, tuning-free Bayesian inference for sparse generalized linear models by introducing EBGLM, a framework that learns the prior within variational inference and optimizes the posterior mean using a real-valued objective . The method employs a mean-field VI with a Taylor-expanded approximate ELBO and establishes a connection to Bayesian normal means via Tweedie’s formula, enabling closed-form updates without sampling. It supports spike-and-slab priors, including Point Normal, Point Laplace, and ash, and can automatically determine the optimal posterior under those priors. Empirical results on simulated sparse logistic regression and real data benchmarks (e.g., 20 Newsgroups and UCI datasets) show improved predictive performance and robustness relative to traditional penalized and variational approaches, highlighting the practical impact of tuning-free, data-driven prior learning in high-dimensional settings.

Abstract

We introduce a flexible empirical Bayes approach for fitting Bayesian generalized linear models. Specifically, we adopt a novel mean-field variational inference (VI) method and the prior is estimated within the VI algorithm, making the method tuning-free. Unlike traditional VI methods that optimize the posterior density function, our approach directly optimizes the posterior mean and prior parameters. This formulation reduces the number of parameters to optimize and enables the use of scalable algorithms such as L-BFGS and stochastic gradient descent. Furthermore, our method automatically determines the optimal posterior based on the prior and likelihood, distinguishing it from existing VI methods that often assume a Gaussian variational. Our approach represents a unified framework applicable to a wide range of exponential family distributions, removing the need to develop unique VI methods for each combination of likelihood and prior distributions. We apply the framework to solve sparse logistic regression and demonstrate the superior predictive performance of our method in extensive numerical studies, by comparing it to prevalent sparse logistic regression approaches.
Paper Structure (22 sections, 2 theorems, 36 equations, 11 figures, 1 table)

This paper contains 22 sections, 2 theorems, 36 equations, 11 figures, 1 table.

Table of Contents

  1. Introduction
  2. Empirical Bayes GLM
  3. Review of variational inference
  4. A novel MFVI formulation
  5. Approximate ELBO
  6. Evaluating approximate ELBO
  7. Choice of priors for sparse GLM
  8. Simulations
  9. Real data Benchmark
  10. Discussion and extensions
  11. Proofs
  12. Proof of Lemma \ref{['lemma:h']}
  13. Proof of Theorem \ref{['thm:penalty']}
  14. Model details
  15. Sparse Gaussian linear regression
  16. ...and 7 more sections

Key Result

Lemma 2.1

Define where $V_{q_{j}}$ is the posterior variance of $\beta_j$ under distribution $q_j$ and $s^2_j(\boldsymbol{\theta}) = a(\phi)/(\sum_i b"(\boldsymbol{x}_i^T\boldsymbol{\theta})x_{ij}^2)$. Then we have where

Figures (11)

  • Figure 1: Average AUC across five simulation settings. We vary (top-left) sample size $n$, (top-right) dimensionality $p$, (middle-left) correlation $\rho$ among features, (middle-right) sparsity level $s$ (number of nonzero coefficients), and (bottom) the distribution of true coefficients. The $y$-axis in each panel shows the mean AUC based on 100 replications, and each line corresponds to one of six competing methods. Overall, the EBGLM-based methods consistently achieve higher AUC in most settings, demonstrating robustness to varying settings. Additional plots are available in the Appendix \ref{['appendix:plots']}.
  • Figure 2: AUC Performance on the 20 Newsgroups Dataset. Each subfigure shows a box plot of AUC scores for different methods when classifying documents in one category versus the rest within that topic. Each box plot summarizes performance across multiple random splits into train/test sets. EBGLM-based methods yield higher median AUC and exhibit less variance (shorter boxes) across these four categories, suggesting both strong predictive performance and robustness in high-dimensional text classification.
  • Figure 3: Comparison of marginal variational posterior densities of regression coefficients in Bayesian logistic regression. Regression coefficients are drawn from standard normal prior $\beta_j\sim N(0,1)$ for $j=1,2,...,10.$
  • Figure 4: Comparison of marginal variational posterior densities and posterior mean of regression coefficients in Bayesian logistic regression. Regression coefficients are drawn from point normal prior $\beta_j\sim 0.5\delta_0+0.5N(0,1)$ for $j=1,2,...,10.$ Correlations among features $\rho=0$.
  • Figure 5: Comparison of marginal variational posterior densities and posterior mean of regression coefficients in Bayesian logistic regression. Regression coefficients are drawn from point normal prior $\beta_j\sim 0.5\delta_0+0.5N(0,1)$ for $j=1,2,...,10.$ Correlations among features $\rho=0.8$.
  • ...and 6 more figures

Theorems & Definitions (4)

  • Lemma 2.1
  • Theorem 2.2
  • proof
  • proof