DyPCL: Dynamic Phoneme-level Contrastive Learning for Dysarthric Speech Recognition

TL;DR

This work tackles dysarthric speech recognition by learning invariant phoneme-level representations. It introduces Dynamic Phoneme-level Contrastive Learning (DyPCL), which combines phoneme-level contrastive learning with a dynamic CTC alignment mechanism and a dynamic curriculum for negative samples, enabling end-to-end training without external aligners. Empirical results on UASpeech show substantial WER reductions, with DyPCL (GP) achieving around 20.23% WER on the TEST set and strong improvements across intelligibility levels, including VL. The approach yields finer-grained phoneme discrimination, as evidenced by embedding clustering, and demonstrates practical impact by outperforming prior state-of-the-art methods on standard benchmarks without data augmentation.

Abstract

Dysarthric speech recognition often suffers from performance degradation due to the intrinsic diversity of dysarthric severity and extrinsic disparity from normal speech. To bridge these gaps, we propose a Dynamic Phoneme-level Contrastive Learning (DyPCL) method, which leads to obtaining invariant representations across diverse speakers. We decompose the speech utterance into phoneme segments for phoneme-level contrastive learning, leveraging dynamic connectionist temporal classification alignment. Unlike prior studies focusing on utterance-level embeddings, our granular learning allows discrimination of subtle parts of speech. In addition, we introduce dynamic curriculum learning, which progressively transitions from easy negative samples to difficult-to-distinguishable negative samples based on phonetic similarity of phoneme. Our approach to training by difficulty levels alleviates the inherent variability of speakers, better identifying challenging speeches. Evaluated on the UASpeech dataset, DyPCL outperforms baseline models, achieving an average 22.10\% relative reduction in word error rate (WER) across the overall dysarthria group.

Figures (7)

• Figure 1: Phoneme-level Contrastive Learning. The phoneme-aligned speech segment corresponding to the phoneme "h" in the word "HOTEL" is used as both the anchor and the positive sample, while the phoneme "f" from the word "ALPHA" serves as the negative sample.
• Figure 2: Phoneme embedding extraction for the target phoneme "h" in the word "HOTEL" using Dynamic CTC Alignment for phoneme-level contrastive learning.
• Figure 3: The illustration of difficulty level determination in three (easy, mid, hard) levels by phoneme distance measurement (Left) and the group-phoneme (GP) curriculum learning (Right). The figure shows an example where the anchor and positive samples are "v".
• Figure 4: UMAP Visualization of Phoneme embeddings on TEST set (VL Group only): Phoneme embeddings, extracted via forced CTC alignment (Figure \ref{['fig:align']}), are shown for three models: CTC, Contrastive learning with random sampling (R), and DyPCL with group-phoneme level curriculum (GP). Points are color-coded by phoneme, illustrating how each model clusters and separates them. For each phoneme, up to 100 embeddings were displayed.
• Figure 5: Distribution of Phoneme distance over phoneme pairs.