Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nash: Neural Adaptive Shrinkage for Structured High-Dimensional Regression

William R. P. Denault

TL;DR

Nash tackles high-dimensional sparse regression with covariate-side information by learning a covariate-conditioned prior on regression effects: $b_j \sim g(d_j,\theta)$, under a Gaussian observation model $y|X,\beta,\sigma^2 \sim N(X\beta,\sigma^2)$. It introduces split variational empirical Bayes (split VEB) to decouple prior learning from posterior inference, enabling efficient coordinate updates and connecting to empirical Bayes regression. The framework unifies group, fused-graph, and beyond-regularization penalties through flexible priors (e.g., mixtures, MDNs), allowing cross-domain adaptation without cross-validation. Empirical results on real datasets and MNIST denoising demonstrate competitive accuracy and scalability, with clear gains when informative side information is available. Overall, Nash provides a flexible, scalable approach to data-driven adaptive regularization in structured, high-dimensional settings.

Abstract

Sparse linear regression is a fundamental tool in data analysis. However, traditional approaches often fall short when covariates exhibit structure or arise from heterogeneous sources. In biomedical applications, covariates may stem from distinct modalities or be structured according to an underlying graph. We introduce Neural Adaptive Shrinkage (Nash), a unified framework that integrates covariate-specific side information into sparse regression via neural networks. Nash adaptively modulates penalties on a per-covariate basis, learning to tailor regularization without cross-validation. We develop a variational inference algorithm for efficient training and establish connections to empirical Bayes regression. Experiments on real data demonstrate that Nash can improve accuracy and adaptability over existing methods.

Nash: Neural Adaptive Shrinkage for Structured High-Dimensional Regression

TL;DR

Nash tackles high-dimensional sparse regression with covariate-side information by learning a covariate-conditioned prior on regression effects: , under a Gaussian observation model . It introduces split variational empirical Bayes (split VEB) to decouple prior learning from posterior inference, enabling efficient coordinate updates and connecting to empirical Bayes regression. The framework unifies group, fused-graph, and beyond-regularization penalties through flexible priors (e.g., mixtures, MDNs), allowing cross-domain adaptation without cross-validation. Empirical results on real datasets and MNIST denoising demonstrate competitive accuracy and scalability, with clear gains when informative side information is available. Overall, Nash provides a flexible, scalable approach to data-driven adaptive regularization in structured, high-dimensional settings.

Abstract

Sparse linear regression is a fundamental tool in data analysis. However, traditional approaches often fall short when covariates exhibit structure or arise from heterogeneous sources. In biomedical applications, covariates may stem from distinct modalities or be structured according to an underlying graph. We introduce Neural Adaptive Shrinkage (Nash), a unified framework that integrates covariate-specific side information into sparse regression via neural networks. Nash adaptively modulates penalties on a per-covariate basis, learning to tailor regularization without cross-validation. We develop a variational inference algorithm for efficient training and establish connections to empirical Bayes regression. Experiments on real data demonstrate that Nash can improve accuracy and adaptability over existing methods.
Paper Structure (31 sections, 1 theorem, 40 equations, 9 figures, 1 table, 1 algorithm)

This paper contains 31 sections, 1 theorem, 40 equations, 9 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Previous works and contribution
  3. Problem definition
  4. Variational Empirical Bayes for the Nash Model
  5. Split Variational inference
  6. Form of the ELBO
  7. High-Level Coordinate Ascent Update for Nash
  8. Choice of $\mathcal{G}$
  9. On the Update for $\beta$
  10. An autoregressive update for $g$ with auto-adaptive dampening
  11. Comparison and Connection with mr.ash
  12. Connection to Penalized Linear Regression and Beyond
  13. Group-Based and Hierarchical Penalty
  14. Fused Lasso and Graph-Based Penalty
  15. Beyond Regularization
  16. ...and 16 more sections

Key Result

Theorem B.1

The profiled objective function $F(q_{ b_j};\sigma^2, \sigma_0^2) = \max_{q_{\beta_j }} F(q_{\beta_j}, q_{ b_j};\sigma^2, \sigma_0^2)_{Nash}$ is a lower bound of $F (q_{b_j}; \sigma^2)_{mr.ash}$.

Figures (9)

  • Figure 1: Upper panel: Adaptation of Figure 1 from kim_flexible_2024, showcasing that posterior mean shrinkage operators (left panel) for different choices of $\sigma_1^2, \ldots ,\sigma_M^2$ and $\pi_0, \ldots, \pi_M$ can mimic the shrinkage operators from some commonly used penalties (right-hand panel). Bottom panel left: Illustration of how Nash can mimic fused Lasso penalty when used with a graph neural net prior-based. The left image presents the induced prior density from equation \ref{['eq:Nash-fused']}, allowing Nash to mimic the fused Lasso penalty ( using $s_1 = 0.45$ and $s_2 = 0.15$). Bottom right panel, penalty surface of the fused Lasso (i.e., $|b_1 +|b_2|+|b_1-b_2|$).
  • Figure 2: Top panel: performances of the different approaches for denoising MNIST image in terms of RMSE. Bottom panel: sample from the experiment showcasing the performance of Nash-fused.
  • Figure 3: Additional denoised image
  • Figure 4: Additional denoised image
  • Figure 5: Additional denoised image
  • ...and 4 more figures

Theorems & Definitions (2)

  • Theorem B.1
  • proof