Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sparse Bayesian Learning Algorithms Revisited: From Learning Majorizers to Structured Algorithmic Learning using Neural Networks

Rushabha Balaji, Kuan-Lin Chen, Danijela Cabric, Bhaskar D. Rao

Abstract

Sparse Bayesian Learning is one of the most popular sparse signal recovery methods, and various algorithms exist under the SBL paradigm. However, given a performance metric and a sparse recovery problem, it is difficult to know a-priori the best algorithm to choose. This difficulty is in part due to a lack of a unified framework to derive SBL algorithms. We address this issue by first showing that the most popular SBL algorithms can be derived using the majorization-minimization (MM) principle, providing hitherto unknown convergence guarantees to this class of SBL methods. Moreover, we show that the two most popular SBL update rules not only fall under the MM framework but are both valid descent steps for a common majorizer, revealing a deeper analytical compatibility between these algorithms. Using this insight and properties from MM theory we expand the class of SBL algorithms, and address finding the best SBL algorithm via data within the MM framework. Second, we go beyond the MM framework by introducing the powerful modeling capabilities of deep learning to further expand the class of SBL algorithms, aiming to learn a superior SBL update rule from data. We propose a novel deep learning architecture that can outperform the classical MM based ones across different sparse recovery problems. Our architecture's complexity does not scale with the measurement matrix dimension, hence providing a unique opportunity to test generalization capability across different matrices. For parameterized dictionaries, this invariance allows us to train and test the model across different parameter ranges. We also showcase our model's ability to learn a functional mapping by its zero-shot performance on unseen measurement matrices. Finally, we test our model's performance across different numbers of snapshots, signal-to-noise ratios, and sparsity levels.

Sparse Bayesian Learning Algorithms Revisited: From Learning Majorizers to Structured Algorithmic Learning using Neural Networks

Abstract

Sparse Bayesian Learning is one of the most popular sparse signal recovery methods, and various algorithms exist under the SBL paradigm. However, given a performance metric and a sparse recovery problem, it is difficult to know a-priori the best algorithm to choose. This difficulty is in part due to a lack of a unified framework to derive SBL algorithms. We address this issue by first showing that the most popular SBL algorithms can be derived using the majorization-minimization (MM) principle, providing hitherto unknown convergence guarantees to this class of SBL methods. Moreover, we show that the two most popular SBL update rules not only fall under the MM framework but are both valid descent steps for a common majorizer, revealing a deeper analytical compatibility between these algorithms. Using this insight and properties from MM theory we expand the class of SBL algorithms, and address finding the best SBL algorithm via data within the MM framework. Second, we go beyond the MM framework by introducing the powerful modeling capabilities of deep learning to further expand the class of SBL algorithms, aiming to learn a superior SBL update rule from data. We propose a novel deep learning architecture that can outperform the classical MM based ones across different sparse recovery problems. Our architecture's complexity does not scale with the measurement matrix dimension, hence providing a unique opportunity to test generalization capability across different matrices. For parameterized dictionaries, this invariance allows us to train and test the model across different parameter ranges. We also showcase our model's ability to learn a functional mapping by its zero-shot performance on unseen measurement matrices. Finally, we test our model's performance across different numbers of snapshots, signal-to-noise ratios, and sparsity levels.

Paper Structure

This paper contains 30 sections, 44 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Notation and Outline
  3. Sparse Bayesian Learning
  4. Classical SBL Algorithms
  5. Expectation-Maximization Algorithm (EM-SBL)
  6. Tipping's Multiplicative Update Rule (MU-SBL)
  7. MM-Based SBL algorithms: Re-interpreting Classical SBL Algorithms
  8. Re-interpreting Classical SBL Algorithms
  9. EM-SBL wipf
  10. MU-SBL tipping
  11. $p$-SBL: A new class of SBL algorithms
  12. Learning the Majorizer via Data
  13. Convex Combination of Update Rules
  14. Convex Combination of Majorizers
  15. Learning an Optimal Convex Combination
  16. ...and 15 more sections

Figures (8)

  • Figure 1: Neural network architecture for the learning new SBL algorithms.
  • Figure 2: The architecture of the $j^{\text{th}}$ iteration. It consists of 6 layers with skip connections. The red dots represent the summation operation.
  • Figure 3: Performance curves of EM, MU-SBL ($p=1$) and $p$-SBL ($p=\{0.25,0.5,0.75\}$) update rules as a function of the number of non-zero elements for the array matrix ($N$=30, $M$=120, SNR=30 dB).
  • Figure 4: Convergence rates of EM, MU-SBL ($p=1$) and $p$-SBL ($p=\{0.25,0.5,0.75\}$) update rules as a function of the number of non-zero elements ($N$=30, $M$=120, SNR=40 dB).
  • Figure 5: Value of the SBL objective function as a function of the number of iterations for different SBL algorithms for the ULA matrix.
  • ...and 3 more figures

Theorems & Definitions (5)

  • proof
  • Definition 1
  • Definition 2
  • proof
  • proof