Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WhisperVC: Decoupled Cross-Domain Alignment and Speech Generation for Low-Resource Whisper-to-Normal Conversion

Dong Liu, Juan Liu, Wei Ju, Yao Tian, Ming Li

Abstract

Whispered speech lacks vocal-fold excitation, making intelligible conversion challenging. We propose WhisperVC, a three-stage framework for low-resource whisper-to-normal (W2N) conversion that decouples cross-domain alignment from speech generation. Stage 1 uses limited paired whisper-normal data with a content encoder and a Conformer-based variational autoencoder (VAE) with soft-DTW alignment to learn domain-invariant semantic representations. Stage 2, trained only on normal speech, employs a Length-Channel Aligner and a two-stage speaker-conditioned mel generator for timbre and prosody modeling. Stage 3 fine-tunes a HiFi-GAN vocoder for waveform synthesis. Experimental results on AISHELL6-Whisper show competitive quality (DNSMOS 3.07, UTMOS 2.83, CER 16.93%) and WavLM speaker similarity (0.95). The framework also supports privacy-preserving communication as well as non-vocal communication and a rehabilitation tool for post-surgical vocal-fold patients. Samples are available online.

WhisperVC: Decoupled Cross-Domain Alignment and Speech Generation for Low-Resource Whisper-to-Normal Conversion

Abstract

Whispered speech lacks vocal-fold excitation, making intelligible conversion challenging. We propose WhisperVC, a three-stage framework for low-resource whisper-to-normal (W2N) conversion that decouples cross-domain alignment from speech generation. Stage 1 uses limited paired whisper-normal data with a content encoder and a Conformer-based variational autoencoder (VAE) with soft-DTW alignment to learn domain-invariant semantic representations. Stage 2, trained only on normal speech, employs a Length-Channel Aligner and a two-stage speaker-conditioned mel generator for timbre and prosody modeling. Stage 3 fine-tunes a HiFi-GAN vocoder for waveform synthesis. Experimental results on AISHELL6-Whisper show competitive quality (DNSMOS 3.07, UTMOS 2.83, CER 16.93%) and WavLM speaker similarity (0.95). The framework also supports privacy-preserving communication as well as non-vocal communication and a rehabilitation tool for post-surgical vocal-fold patients. Samples are available online.

Paper Structure

This paper contains 14 sections, 11 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Framework Overview
  4. Whisper-Specific Domain Alignment
  5. Coarse-to-Fine Residual Generation
  6. Vocoder Adaptation
  7. Experimental Results
  8. Experimental Setup
  9. Evaluation Metrics
  10. Mandarin W2N Results and Ablation Analysis
  11. Voice Conversion Capability (Mandarin)
  12. English W2N Evaluation
  13. Conclusion
  14. Generative AI Use Disclosure

Figures (2)

  • Figure 1: Overview of the proposed whisper-to-normal voice conversion framework.
  • Figure 2: Overview of the proposed Conformer-based VAE module.