Parametric Mean-Field empirical Bayes in high-dimensional linear regression

TL;DR

This work develops a precise asymptotic theory for parametric empirical Bayes in high-dimensional Bayesian linear regression with random design, focusing on a variational EB (vEB) estimator that maximizes a mean-field lower bound of the marginal likelihood. It reveals a phase transition: in the regime $p=o(n^{2/3})$ the vEB estimator achieves $\,\sqrt{p}$-consistency and information-theoretic efficiency, while for larger $p$ it becomes suboptimal, though debiasing can restore a sharper normal limit up to $p\ll n^{3/4}$. The authors further show that EB-based posterior inference closely tracks the oracle posterior for coordinates and certain linear projections in the favorable regime, and they provide calibrated EB-adjusted intervals to achieve valid marginal coverage. Extensive simulations confirm the phase transition, effectiveness of prior recovery, and the utility of debiasing for improved finite-sample performance. Overall, the paper advances understanding of vEB methods in high-dimensional regression, with practical implications for estimating effect-size distributions and downstream uncertainty quantification.

Abstract

In this paper, we consider the problem of parametric empirical Bayes estimation of an i.i.d. prior in high-dimensional Bayesian linear regression, with random design. We obtain the asymptotic distribution of the variational Empirical Bayes (vEB) estimator, which approximately maximizes a variational lower bound of the intractable marginal likelihood. We characterize a sharp phase transition behavior for the vEB estimator -- namely that it is information theoretically optimal (in terms of limiting variance) up to $p=o(n^{2/3})$ while it suffers from a sub-optimal convergence rate in higher dimensions. In the first regime, i.e., when $p=o(n^{2/3})$, we show how the estimated prior can be calibrated to enable valid coordinate-wise and delocalized inference, both under the \emph{empirical Bayes posterior} and the oracle posterior. In the second regime, we propose a debiasing technique as a way to improve the performance of the vEB estimator beyond $p=o(n^{2/3})$. Extensive numerical experiments corroborate our theoretical findings.

Figures (2)

• Figure 1: Visualization of $\kappa(\theta)$ under various priors: (first) $\textsf{Ber}(\theta)$, (second) Normal with unknown variance: $N(0, 1/\theta)$, (third/fourth) Gaussian mixtures $\frac{1}{2} N(\theta_1,1) + \frac{1}{2} N(\theta_2,0.25)$ with $\theta_2 = -1$ fixed throughout the visualization. For the mixture prior, $\kappa(\theta) = (\kappa_1(\theta), \kappa_2(\theta))$ is a two-dimensional vector, and each component is plotted in the third/fourth figure. We see that the singularity $\kappa(\theta) = 0$ is atypical. In particular, for the Gaussian mixture case, the singularity occurs when the two mixture components share a common mean $\theta_1 = \theta_2 = -1$.
• Figure 2: (Top row) Histogram of the estimators $\hat{\theta}^{\mathsf{vEB}}$ and $\tilde{\theta}^{\mathsf{d}}$, where the prior is the symmetric Gaussian mixture $\frac{1}{2}N(\theta,1)+\frac{1}{2}N(-\theta,1)$. We fix $n = 1000$, and consider $p=\sqrt{n}$ and $n^{2/3}$ in each row. The solid red line shows the true value $\theta_0 = 1$ and the dotted blue line denotes the average across 400 replications (under a fixed realization of $X$). We see that $\hat{\theta}^{\mathsf{vEB}}$ has a positive bias and is right-skewed whereas the debiased estimator $\tilde{\theta}^{\mathsf{d}}$ has smaller bias. (Bottom row) Average bias and variance of $\hat{\theta}^{\mathsf{vEB}}$ and $\tilde{\theta}^{\mathsf{d}}$, where $p$ varies from $10$ to $250$ and $n=1000$ is fixed. As $p$ increases, the bias of $\hat{\theta}_{\mathnormal p}$ also increases.

Theorems & Definitions (75)

• Theorem
• Definition 1.1: Linear-quadratic tilt
• Definition 2.1: Transformed variables
• Definition 2.2: Mean-Field optimizers
• Remark 2.1: Unknown variance
• Definition 3.1
• Lemma 3.1: Fisher information
• Remark 3.1: On Conditions R4 and R5
• Remark 3.2: On Condition R6
• Lemma 3.2