Bayesian Analysis for Over-parameterized Linear Model via Effective Spectra
Tomoya Wakayama, Masaaki Imaizumi
TL;DR
The paper tackles high-dimensional linear regression with $p \gg n$ under non-sparse $\boldsymbol{\theta}^*$ by introducing a data-adaptive Gaussian prior concentrated on the leading eigen-directions of the covariate covariance. A hierarchical prior selects the effective rank $k$, and a Bernstein–von Mises-type truncation yields a Gaussian approximation to the posterior that facilitates uncertainty quantification with reduced computation. Theoretical contributions include posterior contraction rates tied to spectral quantities (effective ranks) and a robust Gaussian-approximation result that holds beyond sub-Gaussian covariates. Empirically, simulations and a real-data analysis demonstrate accurate prediction and well-calibrated uncertainty, illustrating the practical value of leveraging spectral information in non-sparse Bayesian high-dimensional settings.
Abstract
In high-dimensional Bayesian statistics, various methods have been developed, including prior distributions that induce parameter sparsity to handle many parameters. Yet, these approaches often overlook the rich spectral structure of the covariate matrix, which can be crucial when true signals are not sparse. To address this gap, we introduce a data-adaptive Gaussian prior whose covariance is aligned with the leading eigenvectors of the sample covariance. This prior design targets the data's intrinsic complexity rather than its ambient dimension by concentrating the parameter search along principal data directions. We establish contraction rates of the corresponding posterior distribution, which reveal how the mass in the spectrum affects the prediction error bounds. Furthermore, we derive a truncated Gaussian approximation to the posterior (i.e., a Bernstein-von Mises-type result), which allows for uncertainty quantification with a reduced computational burden. Our findings demonstrate that Bayesian methods leveraging spectral information of the data are effective for estimation in non-sparse, high-dimensional settings.