Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sparsity via Hyperpriors: A Theoretical and Algorithmic Study under Empirical Bayes Framework

Zhitao Li, Yiqiu Dong, Xueying Zeng

TL;DR

The paper analyzes how hyperpriors in the empirical Bayes framework influence sparsity and local optimality for sparse inverse problems. It establishes a theoretical link between hyperprior structure and KKT-point sparsity, showing that half-Laplace and certain half-generalized Gaussian priors promote sparsity and robustness to noise. A PALM-based algorithm with convergence guarantees is proposed and validated on 2D image deblurring, demonstrating improved sparsity and restoration under varying ill-posedness and noise. The work provides both theoretical insights and a practical solver for hyperparameter estimation within EB, with implications for stability and reconstruction quality in sparse Bayesian learning.

Abstract

This paper presents a comprehensive analysis of hyperparameter estimation within the empirical Bayes framework (EBF) for sparse learning. By studying the influence of hyperpriors on the solution of EBF, we establish a theoretical connection between the choice of the hyperprior and the sparsity as well as the local optimality of the resulting solutions. We show that some strictly increasing hyperpriors, such as half-Laplace and half-generalized Gaussian with the power in $(0,1)$, effectively promote sparsity and improve solution stability with respect to measurement noise. Based on this analysis, we adopt a proximal alternating linearized minimization (PALM) algorithm with convergence guaranties for both convex and concave hyperpriors. Extensive numerical tests on two-dimensional image deblurring problems demonstrate that introducing appropriate hyperpriors significantly promotes the sparsity of the solution and enhances restoration accuracy. Furthermore, we illustrate the influence of the noise level and the ill-posedness of inverse problems to EBF solutions.

Sparsity via Hyperpriors: A Theoretical and Algorithmic Study under Empirical Bayes Framework

TL;DR

The paper analyzes how hyperpriors in the empirical Bayes framework influence sparsity and local optimality for sparse inverse problems. It establishes a theoretical link between hyperprior structure and KKT-point sparsity, showing that half-Laplace and certain half-generalized Gaussian priors promote sparsity and robustness to noise. A PALM-based algorithm with convergence guarantees is proposed and validated on 2D image deblurring, demonstrating improved sparsity and restoration under varying ill-posedness and noise. The work provides both theoretical insights and a practical solver for hyperparameter estimation within EB, with implications for stability and reconstruction quality in sparse Bayesian learning.

Abstract

This paper presents a comprehensive analysis of hyperparameter estimation within the empirical Bayes framework (EBF) for sparse learning. By studying the influence of hyperpriors on the solution of EBF, we establish a theoretical connection between the choice of the hyperprior and the sparsity as well as the local optimality of the resulting solutions. We show that some strictly increasing hyperpriors, such as half-Laplace and half-generalized Gaussian with the power in , effectively promote sparsity and improve solution stability with respect to measurement noise. Based on this analysis, we adopt a proximal alternating linearized minimization (PALM) algorithm with convergence guaranties for both convex and concave hyperpriors. Extensive numerical tests on two-dimensional image deblurring problems demonstrate that introducing appropriate hyperpriors significantly promotes the sparsity of the solution and enhances restoration accuracy. Furthermore, we illustrate the influence of the noise level and the ill-posedness of inverse problems to EBF solutions.

Paper Structure

This paper contains 20 sections, 5 theorems, 86 equations, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Empirical Bayes framework
  3. Sparsity and local optimality
  4. Derivatives of $\mathcal{J}(\boldsymbol{\gamma})$
  5. Sparsity and local optimality of KKT points
  6. General Gamma prior
  7. Numerical algorithm
  8. Proximal alternating linearized minimization
  9. Solving the $\boldsymbol{\gamma}$-subproblem
  10. Convergence analysis
  11. Numerical experiments
  12. Impact of different hyperpriors
  13. Comparison of the half-Laplace and half-Gaussian hyperpriors
  14. Influence of the half-generalized Gaussian hyperpriors
  15. Influence of the Gamma hyperprior
  16. ...and 5 more sections

Key Result

Theorem 3.1

Assume $\mathcal{H}$ is nonnegative in $[0,+\infty)$. Then the following statements hold:

Figures (8)

  • Figure 1: The Cameraman image. Top row: test images; Bottom row: absolute values of DCT coefficients.
  • Figure 2: Comparisons of the results using the half-Laplace, half-Gaussian and no hyperprior. Top row: restored images; Bottom row: absolute values of DCT coefficients.
  • Figure 3: Comparisons of the results using the half-generalized Gaussian hyperpriors with different $p$. Top row: restored images; Bottom row: absolute values of DCT coefficients.
  • Figure 4: Comparisons of the results using the Gamma hyperprior with different $\alpha$. Top row: restored images with relative errors of $0.1084$, $0.1055$ and $0.1423$, respectively; Bottom row: absolute values of DCT coefficients with sparsity rates of $94,74\%$, $84.73\%$ and $0\%$, respectively.
  • Figure 5: The compressed House image and its DCT coefficients.
  • ...and 3 more figures

Theorems & Definitions (17)

  • Theorem 3.1
  • Proof 1
  • Corollary 3.2
  • Proof 2
  • Remark 3.3
  • Theorem 3.4
  • Proof 3
  • Remark 3.5
  • Example 3.1
  • Example 3.2
  • ...and 7 more