Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Whispered Speech Recognition Performance using Pseudo-whispered based Data Augmentation

Zhaofeng Lin, Tanvina Patel, Odette Scharenborg

TL;DR

Whispered speech poses a substantial ASR challenge due to distinct acoustic properties and limited training data. The paper introduces a two-step handcrafted DSP pipeline that converts normal speech into pseudo-whispered speech by first removing glottal influence and then modifying formants, using the WORLD vocoder for re-synthesis. Augmenting End-to-End ASR models with pseudo-whispered data from TIMIT, wTIMIT-n, and LibriSpeech achieves up to 18.2% relative WER reduction on whispered speech, with US English accents showing the strongest gains. The analysis reveals that the absence of glottal information is the primary contributor to whispered ASR degradation, highlighting the practical value of pseudo-whisper data for improving ASR in whisper-like contexts with limited real whispered data.

Abstract

Whispering is a distinct form of speech known for its soft, breathy, and hushed characteristics, often used for private communication. The acoustic characteristics of whispered speech differ substantially from normally phonated speech and the scarcity of adequate training data leads to low automatic speech recognition (ASR) performance. To address the data scarcity issue, we use a signal processing-based technique that transforms the spectral characteristics of normal speech to those of pseudo-whispered speech. We augment an End-to-End ASR with pseudo-whispered speech and achieve an 18.2% relative reduction in word error rate for whispered speech compared to the baseline. Results for the individual speaker groups in the wTIMIT database show the best results for US English. Further investigation showed that the lack of glottal information in whispered speech has the largest impact on whispered speech ASR performance.

Improving Whispered Speech Recognition Performance using Pseudo-whispered based Data Augmentation

TL;DR

Whispered speech poses a substantial ASR challenge due to distinct acoustic properties and limited training data. The paper introduces a two-step handcrafted DSP pipeline that converts normal speech into pseudo-whispered speech by first removing glottal influence and then modifying formants, using the WORLD vocoder for re-synthesis. Augmenting End-to-End ASR models with pseudo-whispered data from TIMIT, wTIMIT-n, and LibriSpeech achieves up to 18.2% relative WER reduction on whispered speech, with US English accents showing the strongest gains. The analysis reveals that the absence of glottal information is the primary contributor to whispered ASR degradation, highlighting the practical value of pseudo-whisper data for improving ASR in whisper-like contexts with limited real whispered data.

Abstract

Whispering is a distinct form of speech known for its soft, breathy, and hushed characteristics, often used for private communication. The acoustic characteristics of whispered speech differ substantially from normally phonated speech and the scarcity of adequate training data leads to low automatic speech recognition (ASR) performance. To address the data scarcity issue, we use a signal processing-based technique that transforms the spectral characteristics of normal speech to those of pseudo-whispered speech. We augment an End-to-End ASR with pseudo-whispered speech and achieve an 18.2% relative reduction in word error rate for whispered speech compared to the baseline. Results for the individual speaker groups in the wTIMIT database show the best results for US English. Further investigation showed that the lack of glottal information in whispered speech has the largest impact on whispered speech ASR performance.
Paper Structure (20 sections, 4 figures, 3 tables)

This paper contains 20 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Datasets
  4. wTIMIT
  5. TIMIT
  6. LibriSpeech
  7. Differences between normal and whispered speech
  8. Proposed pseudo-whispered speech conversion
  9. Step 1: Removing glottal information
  10. Step 2: Changing formant information
  11. Experimental setup
  12. Exp. 1: Baseline models
  13. Exp. 2: Pseudo-whisper data augmentation
  14. Exp. 3: Acoustic characteristics of whispered speech
  15. Results
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Spectrograms of the sentence "I gave them several choices and let them set the priorities." produced by the same speaker in a normal and whispered voice. Example is taken from the wTIMIT corpus.
  • Figure 2: The proposed pipeline for pseudo-whispered speech conversion, where GFM-IAIF-GC is GFM-IAIF-based glottal cancellation and MAF is moving average filtering. Input is normal speech and the output is pseudo-whispered speech (PW).
  • Figure 3: Spectrogram of normal (left panel), whispered (middle panel), and pseudo-whispered speech (right panel) of the word "priorities" from the same utterance as in Figure 1.
  • Figure 4: The pipeline for generating speech with only glottal cancellation (top panel) and with only a widened formant bandwidth and shifted formant frequencies (bottom panel).