Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction

Xueyuan Chen, Dongchao Yang, Dingdong Wang, Xixin Wu, Zhiyong Wu, Helen Meng

TL;DR

CoLM-DSR tackles the dysarthric speech reconstruction problem by enhancing speaker similarity and prosody naturalness through a neural codec language modeling framework. It integrates a multi-modal content encoder, a novel speaker codec encoder, and a codec LM-based speech decoder to reconstruct normal speech conditioned on dysarthric input, leveraging large-scale normal speech data for zero-shot voice cloning. The approach uses an audio-visual content stream (via AV-HuBERT) and a normalizing codec pathway (via EnCodec-based tokenizer and GE2E-trained normalizer) to produce speaker-consistent, natural-sounding reconstructions, with AR and NAR stages coordinating quantized codec tokens. Experimental results on UASpeech demonstrate significant improvements in both subjective and objective metrics for speaker similarity, naturalness, and intelligibility, underscoring the practical potential for assistive communication and low-resource adaptation in DSR.

Abstract

Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech into normal speech. It still suffers from low speaker similarity and poor prosody naturalness. In this paper, we propose a multi-modal DSR model by leveraging neural codec language modeling to improve the reconstruction results, especially for the speaker similarity and prosody naturalness. Our proposed model consists of: (i) a multi-modal content encoder to extract robust phoneme embeddings from dysarthric speech with auxiliary visual inputs; (ii) a speaker codec encoder to extract and normalize the speaker-aware codecs from the dysarthric speech, in order to provide original timbre and normal prosody; (iii) a codec language model based speech decoder to reconstruct the speech based on the extracted phoneme embeddings and normalized codecs. Evaluations on the commonly used UASpeech corpus show that our proposed model can achieve significant improvements in terms of speaker similarity and prosody naturalness.

CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction

TL;DR

CoLM-DSR tackles the dysarthric speech reconstruction problem by enhancing speaker similarity and prosody naturalness through a neural codec language modeling framework. It integrates a multi-modal content encoder, a novel speaker codec encoder, and a codec LM-based speech decoder to reconstruct normal speech conditioned on dysarthric input, leveraging large-scale normal speech data for zero-shot voice cloning. The approach uses an audio-visual content stream (via AV-HuBERT) and a normalizing codec pathway (via EnCodec-based tokenizer and GE2E-trained normalizer) to produce speaker-consistent, natural-sounding reconstructions, with AR and NAR stages coordinating quantized codec tokens. Experimental results on UASpeech demonstrate significant improvements in both subjective and objective metrics for speaker similarity, naturalness, and intelligibility, underscoring the practical potential for assistive communication and low-resource adaptation in DSR.

Abstract

Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech into normal speech. It still suffers from low speaker similarity and poor prosody naturalness. In this paper, we propose a multi-modal DSR model by leveraging neural codec language modeling to improve the reconstruction results, especially for the speaker similarity and prosody naturalness. Our proposed model consists of: (i) a multi-modal content encoder to extract robust phoneme embeddings from dysarthric speech with auxiliary visual inputs; (ii) a speaker codec encoder to extract and normalize the speaker-aware codecs from the dysarthric speech, in order to provide original timbre and normal prosody; (iii) a codec language model based speech decoder to reconstruct the speech based on the extracted phoneme embeddings and normalized codecs. Evaluations on the commonly used UASpeech corpus show that our proposed model can achieve significant improvements in terms of speaker similarity and prosody naturalness.
Paper Structure (14 sections, 6 equations, 3 figures, 3 tables)

This paper contains 14 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Multi-modal Encoder for Content Extraction
  4. Speaker Codec Encoder for Timbre Preservation and Prosody Normalization
  5. Codec LM based Speech Decoder for Speech Reconstruction
  6. Experiments
  7. Experimental Settings
  8. Experimental Results
  9. Speaker Similarity Comparison
  10. Speech Naturalness Comparison
  11. Speech Intelligibility Comparison
  12. Investigation on Dysarthric Codecs and Normal Codecs
  13. Conclusion
  14. Acknowledgements

Figures (3)

  • Figure 1: Overall architecture of the proposed CoLM-DSR system with Multi-modal Content Encoder, Speaker Codec Encoder and Codec LM based Speech Decoder.
  • Figure 2: Diagram of the proposed CoLM-DSR system, where (a), (b) and (c) show the model details of Multi-modal Encoder, Speech Codec Encoder and Neural Codec Language Modeling respectively.
  • Figure 3: Results of AB Preference Test.