Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploiting Audio-Visual Features with Pretrained AV-HuBERT for Multi-Modal Dysarthric Speech Reconstruction

Xueyuan Chen, Yuejiao Wang, Xixin Wu, Disong Wang, Zhiyong Wu, Xunying Liu, Helen Meng

TL;DR

This work addresses dysarthric speech reconstruction by introducing a multi-modal framework that exploits audio-visual cues, notably lip movements, to enhance phoneme representation extraction. It compares three encoder variants (audio-only, VGG-based AV, and AV-HuBERT-based AV) within a unified architecture that also includes a variance adaptor for prosody and a speaker-conditioned mel-decoder. The AV-HuBERT-based audio-visual encoder provides the strongest gains, delivering substantial improvements in WER and MOS on the UASpeech dataset, especially for speakers with severe dysarthria (e.g., up to 42.1% absolute WER reduction relative to original). The results demonstrate the value of visual information and pretrained audio-visual representations for robust DSR in challenging, noisy environments and point toward practical, multi-modal assistive technologies.

Abstract

Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech into normal speech by improving the intelligibility and naturalness. This is a challenging task especially for patients with severe dysarthria and speaking in complex, noisy acoustic environments. To address these challenges, we propose a novel multi-modal framework to utilize visual information, e.g., lip movements, in DSR as extra clues for reconstructing the highly abnormal pronunciations. The multi-modal framework consists of: (i) a multi-modal encoder to extract robust phoneme embeddings from dysarthric speech with auxiliary visual features; (ii) a variance adaptor to infer the normal phoneme duration and pitch contour from the extracted phoneme embeddings; (iii) a speaker encoder to encode the speaker's voice characteristics; and (iv) a mel-decoder to generate the reconstructed mel-spectrogram based on the extracted phoneme embeddings, prosodic features and speaker embeddings. Both objective and subjective evaluations conducted on the commonly used UASpeech corpus show that our proposed approach can achieve significant improvements over baseline systems in terms of speech intelligibility and naturalness, especially for the speakers with more severe symptoms. Compared with original dysarthric speech, the reconstructed speech achieves 42.1\% absolute word error rate reduction for patients with more severe dysarthria levels.

Exploiting Audio-Visual Features with Pretrained AV-HuBERT for Multi-Modal Dysarthric Speech Reconstruction

TL;DR

This work addresses dysarthric speech reconstruction by introducing a multi-modal framework that exploits audio-visual cues, notably lip movements, to enhance phoneme representation extraction. It compares three encoder variants (audio-only, VGG-based AV, and AV-HuBERT-based AV) within a unified architecture that also includes a variance adaptor for prosody and a speaker-conditioned mel-decoder. The AV-HuBERT-based audio-visual encoder provides the strongest gains, delivering substantial improvements in WER and MOS on the UASpeech dataset, especially for speakers with severe dysarthria (e.g., up to 42.1% absolute WER reduction relative to original). The results demonstrate the value of visual information and pretrained audio-visual representations for robust DSR in challenging, noisy environments and point toward practical, multi-modal assistive technologies.

Abstract

Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech into normal speech by improving the intelligibility and naturalness. This is a challenging task especially for patients with severe dysarthria and speaking in complex, noisy acoustic environments. To address these challenges, we propose a novel multi-modal framework to utilize visual information, e.g., lip movements, in DSR as extra clues for reconstructing the highly abnormal pronunciations. The multi-modal framework consists of: (i) a multi-modal encoder to extract robust phoneme embeddings from dysarthric speech with auxiliary visual features; (ii) a variance adaptor to infer the normal phoneme duration and pitch contour from the extracted phoneme embeddings; (iii) a speaker encoder to encode the speaker's voice characteristics; and (iv) a mel-decoder to generate the reconstructed mel-spectrogram based on the extracted phoneme embeddings, prosodic features and speaker embeddings. Both objective and subjective evaluations conducted on the commonly used UASpeech corpus show that our proposed approach can achieve significant improvements over baseline systems in terms of speech intelligibility and naturalness, especially for the speakers with more severe symptoms. Compared with original dysarthric speech, the reconstructed speech achieves 42.1\% absolute word error rate reduction for patients with more severe dysarthria levels.
Paper Structure (15 sections, 4 figures, 1 table)

This paper contains 15 sections, 4 figures, 1 table.

Table of Contents

  1. introduction
  2. methodology
  3. Multi-modal Encoder for Phoneme Embeddings Extraction
  4. Audio-only Encoder
  5. VGG-based Audio-Visual Encoder
  6. AVHuBERT-based Audio-Visual Encoder
  7. Variance Adaptor for Explicit Prosody Modeling
  8. Speaker Encoder and Mel-Decoder
  9. Experiments
  10. Experimental Settings
  11. Experimental Results and Analysis
  12. Multi-modal Encoder Comparison with Audio-only Encoder
  13. Objective Speech Intelligibility Comparison
  14. Subjective Speech Naturalness Comparison
  15. Conclusion

Figures (4)

  • Figure 1: Diagram of the proposed multi-modal DSR system, where (a) illustrates the overall architecture with Multi-modal Encoder, Variance Adaptor, Speaker Encoder and Mel-Decoder, (b) and (c) show two different Multi-modal Encoders to be compared respectively.
  • Figure 2: Audio-only Encoder
  • Figure 3: Example video snapshots of four UASpeech speakers M12, F02, M16 and F04 with various angles facing the camera.
  • Figure 4: Comparison results of MOS with 95% confidence in terms of speech intelligibility and naturalness.