Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hyperparameter Estimation for Sparse Bayesian Learning Models

Feng Yu, Lixin Shen, Guohui Song

TL;DR

The paper tackles hyperparameter estimation in Sparse Bayesian Learning (SBL) by introducing a unified alternating minimization and linearization (AML) framework that recasts the classical EM, MacKay (MK), and convex bounding (CB) algorithms as linearized surrogates of the nonconvex marginal likelihood objective. It develops a novel AML-based algorithm and strengthens it with an AMQ (AM with Quadratic) paradigm that adds proximal regularization to improve convergence, especially at low signal-to-noise ratios. The authors provide convergence analyses and thorough numerical experiments, demonstrating faster convergence and robustness across denoising and general linear inverse problems, including real data from EEG and SAR imagery. This framework offers a flexible, efficient pathway for hyperparameter estimation in SBL with practical impact on denoising and inverse problems demanding sparse solutions.

Abstract

Sparse Bayesian Learning (SBL) models are extensively used in signal processing and machine learning for promoting sparsity through hierarchical priors. The hyperparameters in SBL models are crucial for the model's performance, but they are often difficult to estimate due to the non-convexity and the high-dimensionality of the associated objective function. This paper presents a comprehensive framework for hyperparameter estimation in SBL models, encompassing well-known algorithms such as the expectation-maximization (EM), MacKay, and convex bounding (CB) algorithms. These algorithms are cohesively interpreted within an alternating minimization and linearization (AML) paradigm, distinguished by their unique linearized surrogate functions. Additionally, a novel algorithm within the AML framework is introduced, showing enhanced efficiency, especially under low signal noise ratios. This is further improved by a new alternating minimization and quadratic approximation (AMQ) paradigm, which includes a proximal regularization term. The paper substantiates these advancements with thorough convergence analysis and numerical experiments, demonstrating the algorithm's effectiveness in various noise conditions and signal-to-noise ratios.

Hyperparameter Estimation for Sparse Bayesian Learning Models

TL;DR

The paper tackles hyperparameter estimation in Sparse Bayesian Learning (SBL) by introducing a unified alternating minimization and linearization (AML) framework that recasts the classical EM, MacKay (MK), and convex bounding (CB) algorithms as linearized surrogates of the nonconvex marginal likelihood objective. It develops a novel AML-based algorithm and strengthens it with an AMQ (AM with Quadratic) paradigm that adds proximal regularization to improve convergence, especially at low signal-to-noise ratios. The authors provide convergence analyses and thorough numerical experiments, demonstrating faster convergence and robustness across denoising and general linear inverse problems, including real data from EEG and SAR imagery. This framework offers a flexible, efficient pathway for hyperparameter estimation in SBL with practical impact on denoising and inverse problems demanding sparse solutions.

Abstract

Sparse Bayesian Learning (SBL) models are extensively used in signal processing and machine learning for promoting sparsity through hierarchical priors. The hyperparameters in SBL models are crucial for the model's performance, but they are often difficult to estimate due to the non-convexity and the high-dimensionality of the associated objective function. This paper presents a comprehensive framework for hyperparameter estimation in SBL models, encompassing well-known algorithms such as the expectation-maximization (EM), MacKay, and convex bounding (CB) algorithms. These algorithms are cohesively interpreted within an alternating minimization and linearization (AML) paradigm, distinguished by their unique linearized surrogate functions. Additionally, a novel algorithm within the AML framework is introduced, showing enhanced efficiency, especially under low signal noise ratios. This is further improved by a new alternating minimization and quadratic approximation (AMQ) paradigm, which includes a proximal regularization term. The paper substantiates these advancements with thorough convergence analysis and numerical experiments, demonstrating the algorithm's effectiveness in various noise conditions and signal-to-noise ratios.
Paper Structure (13 sections, 12 theorems, 118 equations, 6 figures)

This paper contains 13 sections, 12 theorems, 118 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Sparse Bayesian Learning Models
  3. A Unified AML Framework for Hyperparameter Estimation
  4. A unified AML framework
  5. EM in the unified AML framework
  6. MK in the unified AML framework
  7. CB in the unified AML framework
  8. Hyperparameter Estimation for Denoising Problem
  9. An AMQ Hyperparameter Estimation Method
  10. Numerical Experiments
  11. Synthetic data for linear inverse problems
  12. Real data: EEG and GOTCHA SAR image
  13. Conclusion

Key Result

Theorem 3.1

\newlabelthm:auxiliary0 For any $\bm{\gamma} \in \mathbb{R}_{+}^{n}$ and $\bm{y}\in \mathbb{R}^{m}$, the optimization problem has a unique minimizer. Denote this minimizer by $\bm{x}^{*}(\bm{\gamma})$. We have that where $\bm{\mu}(\bm{\gamma})$ is given by eq:mu_Sigma_gamma and $\mathsf{S}(\bm{\gamma})$ is defined by eq:evidence.

Figures (6)

  • Figure 1: Comparison of convergence rates.
  • Figure 1: Approximation errors for the denoising problem. Top row: $s=10$, bottom row: $s=80$. Left column: $\beta=10^{-1}$, middle column: $\beta=1$, right column: $\beta=10$.
  • Figure 2: Objective function values for the Fourier reconstruction problem. Top row: $s=10$, bottom row: $s=80$. Left column: $\beta=10^{-1}$, middle column: $\beta=1$, right column: $\beta=10$.
  • Figure 3: Objective function values for the Fourier reconstruction problem with different $\tau$ values. Top row: $s=10$, bottom row: $s=80$. Left column: $\beta=10^{-1}$, middle column: $\beta=1$, right column: $\beta=10$.
  • Figure 4: Convergence comparison in EEG.
  • ...and 1 more figures

Theorems & Definitions (24)

  • Theorem 3.1
  • Proof 1
  • Proposition 3.2
  • Proof 2
  • Proposition 3.3
  • Proof 3
  • Proposition 3.4
  • Proof 4
  • Proposition 4.1
  • Proof 5
  • ...and 14 more