Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

Zengrui Jin, Mengzhe Geng, Jiajun Deng, Tianzi Wang, Shujie Hu, Guinan Li, Xunying Liu

TL;DR

This work tackles ASR for dysarthric and elderly speech by addressing data scarcity with speaker-dependent adversarial data augmentation. It introduces two GAN-based strategies: parallel-data DCGANs that transform tempo/speed perturbed control speech to target impairments, and non-parallel data spectral-basis GANs that perturb SVD-derived spectral bases before reassembly with temporal components. Across four datasets and both TDNN and Conformer backends, the proposed methods yield consistent WER/CER improvements over strong baselines and retain gains after LHUC-SAT adaptation, demonstrating the approach's effectiveness and generality. The results suggest that modeling fine-grained spectro-temporal differences via personalized GANs can substantially enhance recognition of dysarthric and elderly speech in practical ASR systems.

Abstract

Despite the rapid progress of automatic speech recognition (ASR) technologies targeting normal speech, accurate recognition of dysarthric and elderly speech remains highly challenging tasks to date. It is difficult to collect large quantities of such data for ASR system development due to the mobility issues often found among these users. To this end, data augmentation techniques play a vital role. In contrast to existing data augmentation techniques only modifying the speaking rate or overall shape of spectral contour, fine-grained spectro-temporal differences between dysarthric, elderly and normal speech are modelled using a novel set of speaker dependent (SD) generative adversarial networks (GAN) based data augmentation approaches in this paper. These flexibly allow both: a) temporal or speed perturbed normal speech spectra to be modified and closer to those of an impaired speaker when parallel speech data is available; and b) for non-parallel data, the SVD decomposed normal speech spectral basis features to be transformed into those of a target elderly speaker before being re-composed with the temporal bases to produce the augmented data for state-of-the-art TDNN and Conformer ASR system training. Experiments are conducted on four tasks: the English UASpeech and TORGO dysarthric speech corpora; the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech datasets. The proposed GAN based data augmentation approaches consistently outperform the baseline speed perturbation method by up to 0.91% and 3.0% absolute (9.61% and 6.4% relative) WER reduction on the TORGO and DementiaBank data respectively. Consistent performance improvements are retained after applying LHUC based speaker adaptation.

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

TL;DR

This work tackles ASR for dysarthric and elderly speech by addressing data scarcity with speaker-dependent adversarial data augmentation. It introduces two GAN-based strategies: parallel-data DCGANs that transform tempo/speed perturbed control speech to target impairments, and non-parallel data spectral-basis GANs that perturb SVD-derived spectral bases before reassembly with temporal components. Across four datasets and both TDNN and Conformer backends, the proposed methods yield consistent WER/CER improvements over strong baselines and retain gains after LHUC-SAT adaptation, demonstrating the approach's effectiveness and generality. The results suggest that modeling fine-grained spectro-temporal differences via personalized GANs can substantially enhance recognition of dysarthric and elderly speech in practical ASR systems.

Abstract

Despite the rapid progress of automatic speech recognition (ASR) technologies targeting normal speech, accurate recognition of dysarthric and elderly speech remains highly challenging tasks to date. It is difficult to collect large quantities of such data for ASR system development due to the mobility issues often found among these users. To this end, data augmentation techniques play a vital role. In contrast to existing data augmentation techniques only modifying the speaking rate or overall shape of spectral contour, fine-grained spectro-temporal differences between dysarthric, elderly and normal speech are modelled using a novel set of speaker dependent (SD) generative adversarial networks (GAN) based data augmentation approaches in this paper. These flexibly allow both: a) temporal or speed perturbed normal speech spectra to be modified and closer to those of an impaired speaker when parallel speech data is available; and b) for non-parallel data, the SVD decomposed normal speech spectral basis features to be transformed into those of a target elderly speaker before being re-composed with the temporal bases to produce the augmented data for state-of-the-art TDNN and Conformer ASR system training. Experiments are conducted on four tasks: the English UASpeech and TORGO dysarthric speech corpora; the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech datasets. The proposed GAN based data augmentation approaches consistently outperform the baseline speed perturbation method by up to 0.91% and 3.0% absolute (9.61% and 6.4% relative) WER reduction on the TORGO and DementiaBank data respectively. Consistent performance improvements are retained after applying LHUC based speaker adaptation.
Paper Structure (26 sections, 7 equations, 4 figures, 7 tables)

This paper contains 26 sections, 7 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Signal Based Dysarthric and Elderly Speech Data Augmentation
  3. Tempo Perturbation
  4. Speed Perturbation
  5. Speaker Dependent Perturbation Factor Estimation
  6. Adversarial Augmentation Using Parallel Data
  7. DCGAN Model Architecture
  8. DCGAN Model Training
  9. Dysarthric Speech Spectrum Generation
  10. Adversarial Augmentation Using Non-Parallel Data
  11. Speech Spectrum Subspace Decomposition
  12. Spectral Basis GAN Model Architecture
  13. Spectral Basis GAN Model Training
  14. Elderly Speech Spectrum Generation
  15. Experiments
  16. ...and 11 more sections

Figures (4)

  • Figure 1: The architectures of the proposed adversarial neural network models designed for impaired or elderly speaker dependent data augmentation using (a) parallel normal and dysarthric speech utterances of identical contents shown in Fig. \ref{['fig:subfig:training']}-\ref{['fig:subfig:conversion']}; and (b) non-parallel normal, non-aged and elderly speech data shown in Fig. \ref{['fig:subfig:training_non_parallel']}-\ref{['fig:subfig:conversion_non_parallel']}.
  • Figure 2: Example spectrogram of (a) control, (b) dysarthric, (c) tempo or (d) speed perturbed control speech, and (e) tempo-GAN or (f) speed-GAN generated speech.
  • Figure 3: Illustration of (a) DCGAN model training on parallel control and dysarthric utterances with modified duration and time alignment; (b) DCGAN based speaker dependent dysarthric speech spectrogram generation using impaired speaker level tempo/speed perturbed normal speech as the input; (c) Spectral basis GAN model training on SVD decomposed non-parallel non-aged and elderly speech spectrograms; and (d) Spectral basis GAN based speaker dependent elderly speech spectrogram generation by re-composition of perturbed non-aged speech derived spectral basis vectors with their temporal bases.
  • Figure 4: Example subspace decomposition of Mel-spectrograms of: (a) a pair of elderly participant (PAR, right middle in (a)) and non-aged clinical investigator (INV, left middle in (a)) utterances of the Cantonese word "苹果 (apple)" produced spectral and temporal basis vectors ($\mathbf{U}$ and $\mathbf{V^{\mathrm{T}}}$) of the JCCOCC MoCA (JCMOCA) xu2021speaker corpus; and (b) a pair of elderly participant (PAR, left middle in (b)) and non-aged clinical investigator (INV, right middle in (b)) utterances of the English word "okay" produced spectral and temporal basis vectors ($\mathbf{U}$ and $\mathbf{V^{\mathrm{T}}}$) of the DementiaBank Pitt (DBANK) becker1994natural dataset.