Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

General Pruning Criteria for Fast SBL

Jakob Möderl, Erik Leitinger, Bernard Henri Fleury

TL;DR

The paper analyzes pruning in Sparse Bayesian Learning (SBL) by examining the marginal likelihood when Gaussian assumptions are weakened to scale-mixture priors (A1–A4). It derives sufficient conditions for hyperparameters to diverge ($\hat{\gamma}_i=\infty$) or remain finite, and shows these conditions are complementary and reduce to the known pruning rule in the Gaussian case (connecting to fast SBL). A graphical interpretation of the pruning mechanism is provided, enhancing intuition about when weights are effectively pruned. The results extend understanding of SBL pruning beyond strictly Gaussian models and relate closely to the pruning behavior of Fast SBL (F-SBL).

Abstract

Sparse Bayesian learning (SBL) associates to each weight in the underlying linear model a hyperparameter by assuming that each weight is Gaussian distributed with zero mean and precision (inverse variance) equal to its associated hyperparameter. The method estimates the hyperparameters by marginalizing out the weights and performing (marginalized) maximum likelihood (ML) estimation. SBL returns many hyperparameter estimates to diverge to infinity, effectively setting the estimates of the corresponding weights to zero (i.e., pruning the corresponding weights from the model) and thereby yielding a sparse estimate of the weight vector. In this letter, we analyze the marginal likelihood as function of a single hyperparameter while keeping the others fixed, when the Gaussian assumptions on the noise samples and the weight distribution that underlies the derivation of SBL are weakened. We derive sufficient conditions that lead, on the one hand, to finite hyperparameter estimates and, on the other, to infinite ones. Finally, we show that in the Gaussian case, the two conditions are complementary and coincide with the pruning condition of fast SBL (F-SBL), thereby providing additional insights into this algorithm.

General Pruning Criteria for Fast SBL

TL;DR

The paper analyzes pruning in Sparse Bayesian Learning (SBL) by examining the marginal likelihood when Gaussian assumptions are weakened to scale-mixture priors (A1–A4). It derives sufficient conditions for hyperparameters to diverge () or remain finite, and shows these conditions are complementary and reduce to the known pruning rule in the Gaussian case (connecting to fast SBL). A graphical interpretation of the pruning mechanism is provided, enhancing intuition about when weights are effectively pruned. The results extend understanding of SBL pruning beyond strictly Gaussian models and relate closely to the pruning behavior of Fast SBL (F-SBL).

Abstract

Sparse Bayesian learning (SBL) associates to each weight in the underlying linear model a hyperparameter by assuming that each weight is Gaussian distributed with zero mean and precision (inverse variance) equal to its associated hyperparameter. The method estimates the hyperparameters by marginalizing out the weights and performing (marginalized) maximum likelihood (ML) estimation. SBL returns many hyperparameter estimates to diverge to infinity, effectively setting the estimates of the corresponding weights to zero (i.e., pruning the corresponding weights from the model) and thereby yielding a sparse estimate of the weight vector. In this letter, we analyze the marginal likelihood as function of a single hyperparameter while keeping the others fixed, when the Gaussian assumptions on the noise samples and the weight distribution that underlies the derivation of SBL are weakened. We derive sufficient conditions that lead, on the one hand, to finite hyperparameter estimates and, on the other, to infinite ones. Finally, we show that in the Gaussian case, the two conditions are complementary and coincide with the pruning condition of fast SBL (F-SBL), thereby providing additional insights into this algorithm.

Paper Structure

This paper contains 9 sections, 3 theorems, 25 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Signal Model and Sparse Bayesian Learning
  3. Analysis of the Marginal Likelihood Under General Scale-Mixture Priors
  4. Classical SBL Example and Graphical Illustration
  5. Classical SBL Example
  6. Graphical Perspective
  7. Conclusion and Outlook
  8. Proof of Lemma \ref{['lemma:marginal-likelihood-moments']}
  9. Application of Theorem \ref{['theorem:divergence']} to the Gaussian Case

Key Result

Lemma 1

Under Assumptions A1-A4 the function $\ell(\gamma)$, see eq:marginal-likelihood-single-gamma, can be written as as $\gamma\uparrow\infty$. In particular, $\lim_{\gamma\uparrow\infty}\ell(\gamma)=f(0)$.

Figures (2)

  • Figure 1: Illustration of the graphs of $f(x)$ and its tangent $t(x)$ at $x=0$. In Case (a) $f(x)\propto\mathrm{N}(x;\,1.5,1)$ which is strictly convex on $I_{0.5}=(-0.5,0.5)$; in Case (b) $f(x)\propto\mathrm{N}(x;\, 0.5,1)$ which is strictly concave on $I_{0.5}$.
  • Figure 2: Graph of the functions $f(x)$, $t(x)$, and $p(x;\gamma)$ for two values $\gamma^{[1]}$ and $\gamma^{[2]}$ of $\gamma$ such that $p(x;\gamma^{[1]})$ has most of its mass concentrated within the interval $I_{0.5}=(-0.5,0.5)$, $\gamma^{[1]} < \gamma^{[2]}$, and $f(x)$ is strictly convex on $I_{0.5}$.

Theorems & Definitions (6)

  • Lemma 1
  • proof
  • Theorem 1
  • proof
  • Theorem 2
  • proof