Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Speech Enhancement Using Burst Propagation

Mohsin Raza, Leandro A. Passos, Ahmed Khubaib, Ahsan Adeel

TL;DR

The MBURST is a novel multimodal solution for audio-visual speech enhancements that consider the most recent neurological discoveries regarding pyramidal cells of the prefrontal cortex and other brain regions, and can reproduce similar mask reconstructions to the multimodal backpropagation-based baseline.

Abstract

This paper proposes the MBURST, a novel multimodal solution for audio-visual speech enhancements that consider the most recent neurological discoveries regarding pyramidal cells of the prefrontal cortex and other brain regions. The so-called burst propagation implements several criteria to address the credit assignment problem in a more biologically plausible manner: steering the sign and magnitude of plasticity through feedback, multiplexing the feedback and feedforward information across layers through different weight connections, approximating feedback and feedforward connections, and linearizing the feedback signals. MBURST benefits from such capabilities to learn correlations between the noisy signal and the visual stimuli, thus attributing meaning to the speech by amplifying relevant information and suppressing noise. Experiments conducted over a Grid Corpus and CHiME3-based dataset show that MBURST can reproduce similar mask reconstructions to the multimodal backpropagation-based baseline while demonstrating outstanding energy efficiency management, reducing the neuron firing rates to values up to \textbf{$70\%$} lower. Such a feature implies more sustainable implementations, suitable and desirable for hearing aids or any other similar embedded systems.

Multimodal Speech Enhancement Using Burst Propagation

TL;DR

The MBURST is a novel multimodal solution for audio-visual speech enhancements that consider the most recent neurological discoveries regarding pyramidal cells of the prefrontal cortex and other brain regions, and can reproduce similar mask reconstructions to the multimodal backpropagation-based baseline.

Abstract

This paper proposes the MBURST, a novel multimodal solution for audio-visual speech enhancements that consider the most recent neurological discoveries regarding pyramidal cells of the prefrontal cortex and other brain regions. The so-called burst propagation implements several criteria to address the credit assignment problem in a more biologically plausible manner: steering the sign and magnitude of plasticity through feedback, multiplexing the feedback and feedforward information across layers through different weight connections, approximating feedback and feedforward connections, and linearizing the feedback signals. MBURST benefits from such capabilities to learn correlations between the noisy signal and the visual stimuli, thus attributing meaning to the speech by amplifying relevant information and suppressing noise. Experiments conducted over a Grid Corpus and CHiME3-based dataset show that MBURST can reproduce similar mask reconstructions to the multimodal backpropagation-based baseline while demonstrating outstanding energy efficiency management, reducing the neuron firing rates to values up to \textbf{} lower. Such a feature implies more sustainable implementations, suitable and desirable for hearing aids or any other similar embedded systems.
Paper Structure (13 sections, 9 equations, 3 figures, 2 tables)

This paper contains 13 sections, 9 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Theoretical Background
  3. Burstpropagation
  4. MBURST
  5. Methodology
  6. Dataset
  7. Data pre-processing
  8. Ideal Binary Mask
  9. Experimental Setup
  10. Experimental Results
  11. Clean Audio Signal Mask Reconstruction
  12. Energy Analysis
  13. Conclusions

Figures (3)

  • Figure 1: Proposed method. The left image depicts the multimodal network with noisy audio and visual inputs. Both channels comprise two layers of burst-based convolutions, followed by flattening and a burst-based dense layer for feature embedding. Finally, such embeddings are concatenated and used to feed another burst-based dense for classification. Right-top and right-bottom frames illustrate the burst-based convolutional and dense layers, respectively. Notice such layers comprise a forward weight matrix (kernel) $\bm{W}$ and a backward weight matrix $\bm{Y}$.
  • Figure 2: Mask reconstructions: (a) Original mask, (b) reconstruction using Backpropagation, and (c) Burstpropagation.
  • Figure 3: Energy efficiency rate over (a) train and (b) test sets.