Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Heteroscedastic Double Bayesian Elastic Net

Masanari Kimura

TL;DR

$HDBEN$ addresses regression under heteroscedasticity in high dimensions by jointly modeling the mean and log-variance with Elastic Net priors on both components. The framework combines a heteroscedastic Gaussian likelihood with hierarchical priors that enforce sparsity and grouping in the mean and variance coefficients, and performs posterior inference via a Gibbs sampler with MH updates. The authors establish posterior concentration, variable selection consistency, and asymptotic normality under mild conditions, and demonstrate via simulations that $HDBEN$ outperforms existing methods, particularly when heteroscedasticity is present and $d$ is large. This work provides a principled approach for simultaneous mean-variance modeling and regularization, improving both estimation and uncertainty quantification in high-dimensional settings.

Abstract

In many practical applications, regression models are employed to uncover relationships between predictors and a response variable, yet the common assumption of constant error variance is frequently violated. This issue is further compounded in high-dimensional settings where the number of predictors exceeds the sample size, necessitating regularization for effective estimation and variable selection. To address this problem, we propose the Heteroscedastic Double Bayesian Elastic Net (HDBEN), a novel framework that jointly models the mean and log-variance using hierarchical Bayesian priors incorporating both $\ell_1$ and $\ell_2$ penalties. Our approach simultaneously induces sparsity and grouping in the regression coefficients and variance parameters, capturing complex variance structures in the data. Theoretical results demonstrate that proposed HDBEN achieves posterior concentration, variable selection consistency, and asymptotic normality under mild conditions which justifying its behavior. Simulation studies further illustrate that HDBEN outperforms existing methods, particularly in scenarios characterized by heteroscedasticity and high dimensionality.

Heteroscedastic Double Bayesian Elastic Net

TL;DR

addresses regression under heteroscedasticity in high dimensions by jointly modeling the mean and log-variance with Elastic Net priors on both components. The framework combines a heteroscedastic Gaussian likelihood with hierarchical priors that enforce sparsity and grouping in the mean and variance coefficients, and performs posterior inference via a Gibbs sampler with MH updates. The authors establish posterior concentration, variable selection consistency, and asymptotic normality under mild conditions, and demonstrate via simulations that outperforms existing methods, particularly when heteroscedasticity is present and is large. This work provides a principled approach for simultaneous mean-variance modeling and regularization, improving both estimation and uncertainty quantification in high-dimensional settings.

Abstract

In many practical applications, regression models are employed to uncover relationships between predictors and a response variable, yet the common assumption of constant error variance is frequently violated. This issue is further compounded in high-dimensional settings where the number of predictors exceeds the sample size, necessitating regularization for effective estimation and variable selection. To address this problem, we propose the Heteroscedastic Double Bayesian Elastic Net (HDBEN), a novel framework that jointly models the mean and log-variance using hierarchical Bayesian priors incorporating both and penalties. Our approach simultaneously induces sparsity and grouping in the regression coefficients and variance parameters, capturing complex variance structures in the data. Theoretical results demonstrate that proposed HDBEN achieves posterior concentration, variable selection consistency, and asymptotic normality under mild conditions which justifying its behavior. Simulation studies further illustrate that HDBEN outperforms existing methods, particularly in scenarios characterized by heteroscedasticity and high dimensionality.

Paper Structure

This paper contains 22 sections, 5 theorems, 72 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Elastic Net Regularization
  4. Bayesian Regularization
  5. Bayesian Elastic Net Prior
  6. Notation and Definitions
  7. Existing Approaches and Limitations
  8. Motivation for Heteroscedastic Double Bayesian Elastic Net
  9. Methodology
  10. Model Specification
  11. Posterior Inference
  12. Theory
  13. Simulation Studies
  14. Experimental Settings
  15. Data Generating Process
  16. ...and 7 more sections

Key Result

Theorem 1

Under Assumptions asm:true_parameter, asm:design_matrix, asm:bounded_design, and asm:prior_support, there exists a sequence $\varepsilon_n \to 0$ as $n \to \infty$ such that

Figures (3)

  • Figure 1: Heteroscedasticity: $\epsilon_i \sim \exp(\bm{X}_i^\top\bm{\gamma})$.
  • Figure 2: Comparison of posteriors.
  • Figure 3: Estimation of $\beta$ and $\gamma$ at different sparsity levels.

Theorems & Definitions (10)

  • Theorem 1
  • proof
  • Corollary 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • proof
  • Theorem 4: Higher‐Order Asymptotic Expansion for the HDBEN Posterior
  • proof