Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalization

Yuejiao Wang, Xixin Wu, Disong Wang, Lingwei Meng, Helen Meng

TL;DR

The paper tackles dysarthric speech reconstruction by replacing cascaded content encoders with discrete speech units derived from self-supervised representations. It introduces a HuBERT-based speech unit normalizer and a Unit HiFi-GAN vocoder to normalize dysarthric content directly into healthy reference units and synthesize waveforms, yielding a simpler yet more effective DSR pipeline. Key contributions include the use of speech units as discrete content representations, a multi-stage fine-tuning strategy for domain adaptation, and a multi-speaker unit-vocoder, achieving substantial improvements in content restoration and intelligibility on the UASpeech corpus. The results demonstrate strong WER reductions and MOS gains, plus robustness to speed and noise, suggesting practical impact for real-world dysarthria communication and potential extension to other textless languages.

Abstract

Dysarthric speech reconstruction (DSR) systems aim to automatically convert dysarthric speech into normal-sounding speech. The technology eases communication with speakers affected by the neuromotor disorder and enhances their social inclusion. NED-based (Neural Encoder-Decoder) systems have significantly improved the intelligibility of the reconstructed speech as compared with GAN-based (Generative Adversarial Network) approaches, but the approach is still limited by training inefficiency caused by the cascaded pipeline and auxiliary tasks of the content encoder, which may in turn affect the quality of reconstruction. Inspired by self-supervised speech representation learning and discrete speech units, we propose a Unit-DSR system, which harnesses the powerful domain-adaptation capacity of HuBERT for training efficiency improvement and utilizes speech units to constrain the dysarthric content restoration in a discrete linguistic space. Compared with NED approaches, the Unit-DSR system only consists of a speech unit normalizer and a Unit HiFi-GAN vocoder, which is considerably simpler without cascaded sub-modules or auxiliary tasks. Results on the UASpeech corpus indicate that Unit-DSR outperforms competitive baselines in terms of content restoration, reaching a 28.2% relative average word error rate reduction when compared to original dysarthric speech, and shows robustness against speed perturbation and noise.

UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalization

TL;DR

The paper tackles dysarthric speech reconstruction by replacing cascaded content encoders with discrete speech units derived from self-supervised representations. It introduces a HuBERT-based speech unit normalizer and a Unit HiFi-GAN vocoder to normalize dysarthric content directly into healthy reference units and synthesize waveforms, yielding a simpler yet more effective DSR pipeline. Key contributions include the use of speech units as discrete content representations, a multi-stage fine-tuning strategy for domain adaptation, and a multi-speaker unit-vocoder, achieving substantial improvements in content restoration and intelligibility on the UASpeech corpus. The results demonstrate strong WER reductions and MOS gains, plus robustness to speed and noise, suggesting practical impact for real-world dysarthria communication and potential extension to other textless languages.

Abstract

Dysarthric speech reconstruction (DSR) systems aim to automatically convert dysarthric speech into normal-sounding speech. The technology eases communication with speakers affected by the neuromotor disorder and enhances their social inclusion. NED-based (Neural Encoder-Decoder) systems have significantly improved the intelligibility of the reconstructed speech as compared with GAN-based (Generative Adversarial Network) approaches, but the approach is still limited by training inefficiency caused by the cascaded pipeline and auxiliary tasks of the content encoder, which may in turn affect the quality of reconstruction. Inspired by self-supervised speech representation learning and discrete speech units, we propose a Unit-DSR system, which harnesses the powerful domain-adaptation capacity of HuBERT for training efficiency improvement and utilizes speech units to constrain the dysarthric content restoration in a discrete linguistic space. Compared with NED approaches, the Unit-DSR system only consists of a speech unit normalizer and a Unit HiFi-GAN vocoder, which is considerably simpler without cascaded sub-modules or auxiliary tasks. Results on the UASpeech corpus indicate that Unit-DSR outperforms competitive baselines in terms of content restoration, reaching a 28.2% relative average word error rate reduction when compared to original dysarthric speech, and shows robustness against speed perturbation and noise.
Paper Structure (13 sections, 3 figures, 4 tables)

This paper contains 13 sections, 3 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. PROPOSED METHOD
  3. HuBERT and speech units
  4. Speech unit normalizer
  5. Multi-stage fine-tuning strategy
  6. Unit HiFi-GAN vocoder
  7. EXPERIMENTAL SETUP
  8. RESULTS
  9. Baselines comparison on content restoration
  10. Ablation study on multi-stage fine-tuning
  11. Robustness study
  12. CONCLUSIONS
  13. ACKNOWLEDGEMENTS

Figures (3)

  • Figure 1: (a) Diagram of the Unit-DSR system. (b) An example of original speech units of different speakers uttering 'bath', and the reconstructed norm units from the speech unit normalizer, which have a high correspondence with the reference speech units.
  • Figure 2: The structure of Unit HiFi-GAN vocoder.
  • Figure 3: Human listening test accuracy of Unit-DSR with (a) different input speed perturbation ratios and (b) various input SNRs.