Dependency-Aware Shrinkage Priors for High Dimensional Regression

TL;DR

This work introduces dependency-aware shrinkage priors (DASP) that embed coefficient dependencies through a correlation matrix $\Omega$ in high-dimensional regression. Grounded in continuous global-local shrinkage, DASP generalizes prior structure to capture joint behavior among coefficients, informed by the design matrix $X$ or robust covariance estimates. Theoretical results show how $\Omega$ modulates posterior precision and shrinkage patterns, while experiments reveal selective gains in parameter recovery for strongly correlated groups, with typically modest improvements in predictive accuracy. The findings advocate for judicious use of prior dependence—beneficial in structured settings but not universally advantageous, and feasible with an automated workflow that estimates $\Omega$ from the design. Overall, DASP provides a principled tool for incorporating prior dependence when inferential goals emphasize accurate recovery of correlated signals rather than wholesale gains in prediction.

Abstract

In high dimensional regression, global local shrinkage priors have gained significant traction for their ability to yield sparse estimates, improve parameter recovery, and support accurate predictive modeling. While recent work has explored increasingly flexible shrinkage prior structures, the role of explicitly modeling dependencies among coefficients remains largely unexplored. In this paper, we investigate whether incorporating such structures into traditional shrinkage priors improves their performance. We introduce dependency-aware shrinkage priors, an extension of continuous shrinkage priors that integrates correlation structures inspired by Zellner's g prior approach. We provide theoretical insights into how dependence alters the prior and posterior structure, and evaluate the method empirically through simulations and real data. We find that modeling dependence can improve parameter recovery when predictors are strongly correlated, but offers only modest gains in predictive accuracy. These findings suggest that prior dependence should be used selectively and guided by the specific inferential goals of the analysis.

Figures (11)

• Figure 1: KL divergence as a function of the correlation parameter $\rho$ for various structures of $\Omega$. The corresponding algebraic forms are provided in Appendix \ref{['appendix']}.
• Figure 2: Monte Carlo approximations of the bivariate joint marginal prior distribution $p(b_1, b_2)$ under different shrinkage priors:Beta Prime (BP), Dirichlet-Laplace (DL), Horseshoe (HS), Normal-Gamma (NG), and R2D2, and for various correlation levels $\rho$. We use default hyperparameters for each prior, with $\sigma = 1$, and $\Omega = (1 - \rho)I + \rho JJ'$, where $J$ is a $2$-dimensional vector of ones.
• Figure 3: Monte Carlo approximations of the unnormalized conditional prior distribution $p(b_1 \mid b_2)$ for varying values of $b_2$ and $\rho$. As both $b_2$ and $\rho$ increase, the distribution shifts toward larger values of $b_1$, indicating reduced shrinkage.
• Figure 4: Implied priors of the effective number of parameters (Equation \ref{['eq:kappa_omega']}) for different shrinkage priors (rows) and correlations (columns). We have used the default hyperparameters for the different shrinkage priors, set $p=100$ covariates, $\sigma = 1$ and $\Omega = (1-\rho)I + \rho JJ'$, where $J$ is a $p$-dimensional vector of ones.
• Figure 5: $\Delta$ELPD for each model under the BAR1 structure across simulation scenarios. Model names represent the difference between the dependency-aware and standard versions, where positive values indicate improved predictive performance of the dependency-aware versions. We have omitted the BP prior due to high variability.

Theorems & Definitions (1)

• Proposition 1