Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transfer Learning-Based Deep Residual Learning for Speech Recognition in Clean and Noisy Environments

Noussaiba Djeffal, Djamel Addou, Hamza Kheddar, Sid Ahmed Selouani

TL;DR

The paper tackles ASR performance deterioration in noisy environments by applying transfer learning to adapt a pretrained ResNet for speech recognition using Mel-frequency features. The source ResNet-50 is fine-tuned into a target speech model with residual blocks and a compact dense head, trained on Aurora-2 to handle clean and multicondition noise. Empirical results show ResNet significantly surpasses CNN and LSTM baselines, achieving 98.94% accuracy in clean and 91.21% in noisy conditions, with informative confusion matrices and WER analyses supporting robustness. The work demonstrates the efficacy of residual-learning-based transfer in noise-robust ASR and points to future gains from Transformer integration and data augmentation.

Abstract

Addressing the detrimental impact of non-stationary environmental noise on automatic speech recognition (ASR) has been a persistent and significant research focus. Despite advancements, this challenge continues to be a major concern. Recently, data-driven supervised approaches, such as deep neural networks, have emerged as promising alternatives to traditional unsupervised methods. With extensive training, these approaches have the potential to overcome the challenges posed by diverse real-life acoustic environments. In this light, this paper introduces a novel neural framework that incorporates a robust frontend into ASR systems in both clean and noisy environments. Utilizing the Aurora-2 speech database, the authors evaluate the effectiveness of an acoustic feature set for Mel-frequency, employing the approach of transfer learning based on Residual neural network (ResNet). The experimental results demonstrate a significant improvement in recognition accuracy compared to convolutional neural networks (CNN) and long short-term memory (LSTM) networks. They achieved accuracies of 98.94% in clean and 91.21% in noisy mode.

Transfer Learning-Based Deep Residual Learning for Speech Recognition in Clean and Noisy Environments

TL;DR

The paper tackles ASR performance deterioration in noisy environments by applying transfer learning to adapt a pretrained ResNet for speech recognition using Mel-frequency features. The source ResNet-50 is fine-tuned into a target speech model with residual blocks and a compact dense head, trained on Aurora-2 to handle clean and multicondition noise. Empirical results show ResNet significantly surpasses CNN and LSTM baselines, achieving 98.94% accuracy in clean and 91.21% in noisy conditions, with informative confusion matrices and WER analyses supporting robustness. The work demonstrates the efficacy of residual-learning-based transfer in noise-robust ASR and points to future gains from Transformer integration and data augmentation.

Abstract

Addressing the detrimental impact of non-stationary environmental noise on automatic speech recognition (ASR) has been a persistent and significant research focus. Despite advancements, this challenge continues to be a major concern. Recently, data-driven supervised approaches, such as deep neural networks, have emerged as promising alternatives to traditional unsupervised methods. With extensive training, these approaches have the potential to overcome the challenges posed by diverse real-life acoustic environments. In this light, this paper introduces a novel neural framework that incorporates a robust frontend into ASR systems in both clean and noisy environments. Utilizing the Aurora-2 speech database, the authors evaluate the effectiveness of an acoustic feature set for Mel-frequency, employing the approach of transfer learning based on Residual neural network (ResNet). The experimental results demonstrate a significant improvement in recognition accuracy compared to convolutional neural networks (CNN) and long short-term memory (LSTM) networks. They achieved accuracies of 98.94% in clean and 91.21% in noisy mode.
Paper Structure (11 sections, 4 figures, 2 tables)

This paper contains 11 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related research
  4. Proposed approach
  5. Description of the source model (before Transfer learning)
  6. Description of the target model (after Transfer learning)
  7. Experiments
  8. Aurora dataset
  9. Experimental results of ResNet
  10. Analysis of results
  11. Conclusion

Figures (4)

  • Figure 1: Proposed scheme: (a) Source model zhang2023transfer (b) Target model.
  • Figure 2: Confusion matrix of multiclass classification.
  • Figure 3: Confusion matrix of binary classification.
  • Figure 4: WER (%) Recognition rates obtained by CNN, LSTM, BiLSTM, and ResNet in clean and noisy mode.