Lightweight and perceptually-guided voice conversion for electro-laryngeal speech

TL;DR

Electro-laryngeal speech suffers from flat pitch, reduced prosody, and noise, hindering intelligibility. The authors adapt a lightweight, fully causal StreamVC for EL→HE voice conversion by removing pitch/energy modules, employing Whisper-based time alignment, and using a two-stage training regime with HE self-supervised pretraining and EL–HE supervised fine-tuning guided by perceptual losses (notably WavLM and human feedback). The best configuration (+WavLM+HF) yields substantial reductions in character error rate and improvements in naturalness, while closely approaching HE ground-truth performance on multiple metrics; prosody generation and intelligibility remain the main bottlenecks. This work demonstrates the feasibility of real-time EL speech rehabilitation with a compact VC architecture and provides insights into loss design and alignment strategies under data-limited conditions, including robustness to noise.

Abstract

Electro-laryngeal (EL) speech is characterized by constant pitch, limited prosody, and mechanical noise, reducing naturalness and intelligibility. We propose a lightweight adaptation of the state-of-the-art StreamVC framework to this setting by removing pitch and energy modules and combining self-supervised pretraining with supervised fine-tuning on parallel EL and healthy (HE) speech data, guided by perceptual and intelligibility losses. Objective and subjective evaluations across different loss configurations confirm their influence: the best model variant, based on WavLM features and human-feedback predictions (+WavLM+HF), drastically reduces character error rate (CER) of EL inputs, raises naturalness mean opinion score (nMOS) from 1.1 to 3.3, and consistently narrows the gap to HE ground-truth speech in all evaluated metrics. These findings demonstrate the feasibility of adapting lightweight voice conversion architectures to EL voice rehabilitation while also identifying prosody generation and intelligibility improvements as the main remaining bottlenecks.

Figures (5)

• Figure 1: Adapted StreamVC architecture for EL-HE voice conversion. It consists of a content encoder, speaker encoder with learnable pooling, and FiLM-conditioned decoder. Training uses mel-spectral reconstruction, adversarial/feature-matching, and additional guided losses (perceptual, human-feedback, intelligibility, and $F_{0}$).
• Figure 2: Time-alignment pipeline for EL--HE pairs: silence filter, PSOLA time-stretching, WEO feature alignment via DTW.
• Figure 3: Normalized (0-1) evaluation metrics (higher is better). (a) Objective metrics comparing GT, EL, baselines (FreeVC & XVC), and our best method (+WavLM+HF). (b) Subjective metrics comparing EL vs. our method with three loss configurations (+WavLM+HF, +WEO+HF, and +BNF+HF).
• Figure 4: Mel-spectogram comparison: (a) EL input, (b) GT speech and (c) VC for model configuration +WavLM+HF.
• Figure 5: CER as a function of SNR (0-25dB) for conversions under quasi-stationary and non-stationary noise conditions. The red line shows the CER obtained from unconverted, noise-free EL speech.