Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explaining Spectrograms in Machine Learning: A Study on Neural Networks for Speech Classification

Jesin James, Balamurali B. T., Binu Abeysinghe, Junchen Liu

TL;DR

This paper investigates what neural networks learn from spectrograms for speech classification, focusing on vowel identity and voiced/unvoiced distinctions. It uses three ResNet-101-based experiments on the LJSpeech corpus and adapts Class Activation Maps (CAMs) to spectrogram inputs to locate frequency regions driving decisions, comparing findings with linguistic formants. The results show high classification accuracy and reveal that networks predominantly rely on formant-like regions, with occasional high-frequency cues; CAMs reveal region-specific cues that align with some linguistic expectations but do not fully explain all misclassifications. The work contributes to interpretability in speech recognition by linking neural activations to known acoustic cues and suggests directions for advanced visualization techniques to further close the gap between learned representations and linguistic knowledge.

Abstract

This study investigates discriminative patterns learned by neural networks for accurate speech classification, with a specific focus on vowel classification tasks. By examining the activations and features of neural networks for vowel classification, we gain insights into what the networks "see" in spectrograms. Through the use of class activation mapping, we identify the frequencies that contribute to vowel classification and compare these findings with linguistic knowledge. Experiments on a American English dataset of vowels showcases the explainability of neural networks and provides valuable insights into the causes of misclassifications and their characteristics when differentiating them from unvoiced speech. This study not only enhances our understanding of the underlying acoustic cues in vowel classification but also offers opportunities for improving speech recognition by bridging the gap between abstract representations in neural networks and established linguistic knowledge

Explaining Spectrograms in Machine Learning: A Study on Neural Networks for Speech Classification

TL;DR

This paper investigates what neural networks learn from spectrograms for speech classification, focusing on vowel identity and voiced/unvoiced distinctions. It uses three ResNet-101-based experiments on the LJSpeech corpus and adapts Class Activation Maps (CAMs) to spectrogram inputs to locate frequency regions driving decisions, comparing findings with linguistic formants. The results show high classification accuracy and reveal that networks predominantly rely on formant-like regions, with occasional high-frequency cues; CAMs reveal region-specific cues that align with some linguistic expectations but do not fully explain all misclassifications. The work contributes to interpretability in speech recognition by linking neural activations to known acoustic cues and suggests directions for advanced visualization techniques to further close the gap between learned representations and linguistic knowledge.

Abstract

This study investigates discriminative patterns learned by neural networks for accurate speech classification, with a specific focus on vowel classification tasks. By examining the activations and features of neural networks for vowel classification, we gain insights into what the networks "see" in spectrograms. Through the use of class activation mapping, we identify the frequencies that contribute to vowel classification and compare these findings with linguistic knowledge. Experiments on a American English dataset of vowels showcases the explainability of neural networks and provides valuable insights into the causes of misclassifications and their characteristics when differentiating them from unvoiced speech. This study not only enhances our understanding of the underlying acoustic cues in vowel classification but also offers opportunities for improving speech recognition by bridging the gap between abstract representations in neural networks and established linguistic knowledge
Paper Structure (19 sections, 2 figures)

This paper contains 19 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Spectrograms and their Significance in Linguistics
  4. Voiced and Unvoiced Speech
  5. Vowel Sounds
  6. Explaining what Deep Neural Networks Learn
  7. Methodology
  8. Database
  9. Experiment Design
  10. Classification Model Training
  11. Class Activation mapping (CAMs)
  12. Results and Discussion
  13. Vowel classification using all Frequency Components
  14. Class Activation Map Analysis
  15. Confusion Matrix Analysis
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Spectrograms of five American English Vowels, which are voiced sounds and unvoiced sound /s/.
  • Figure 2: CAMs and Confusion Matrix for classifying five vowels with maximum frequency (Sampled at 22050 Hz)).