Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WhispEar: A Bi-directional Framework for Scaling Whispered Speech Conversion via Pseudo-Parallel Whisper Generation

Zihao Fang, Yingda Shen, Zifan Guan, Tongtong Song, Zhenyi Liu, Zhizheng Wu

TL;DR

WhispEar, a bidirectional framework based on unified semantic representations that capture speaking-mode-invariant information shared by whispered and normal speech, is proposed, which contains both W2N and normal-to-whisper (N2W) models.

Abstract

Whispered speech lacks vocal fold vibration and fundamental frequency, resulting in degraded acoustic cues and making whisper-to-normal (W2N) conversion challenging, especially with limited parallel data. We propose WhispEar, a bidirectional framework based on unified semantic representations that capture speaking-mode-invariant information shared by whispered and normal speech. The framework contains both W2N and normal-to-whisper (N2W) models. Notably, the N2W model enables zero-shot pseudo-parallel whisper generation from abundant normal speech, allowing scalable data augmentation for W2N training. Increasing generated data consistently improves performance. We also release the largest bilingual (Chinese-English) whispered-normal parallel corpus to date. Experiments demonstrate that WhispEar outperforms strong baselines and benefits significantly from scalable pseudo-parallel data.

WhispEar: A Bi-directional Framework for Scaling Whispered Speech Conversion via Pseudo-Parallel Whisper Generation

TL;DR

WhispEar, a bidirectional framework based on unified semantic representations that capture speaking-mode-invariant information shared by whispered and normal speech, is proposed, which contains both W2N and normal-to-whisper (N2W) models.

Abstract

Whispered speech lacks vocal fold vibration and fundamental frequency, resulting in degraded acoustic cues and making whisper-to-normal (W2N) conversion challenging, especially with limited parallel data. We propose WhispEar, a bidirectional framework based on unified semantic representations that capture speaking-mode-invariant information shared by whispered and normal speech. The framework contains both W2N and normal-to-whisper (N2W) models. Notably, the N2W model enables zero-shot pseudo-parallel whisper generation from abundant normal speech, allowing scalable data augmentation for W2N training. Increasing generated data consistently improves performance. We also release the largest bilingual (Chinese-English) whispered-normal parallel corpus to date. Experiments demonstrate that WhispEar outperforms strong baselines and benefits significantly from scalable pseudo-parallel data.
Paper Structure (20 sections, 4 equations, 3 figures, 3 tables)

This paper contains 20 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Stage 1: Semantic Tokenizer Distillation
  4. Stage 2: Shared Flow-Matching Acoustic Model
  5. Stage 3: Unified Tokenizer Training and Scalable Expansion
  6. Dataset
  7. Data Collection
  8. Data Processing
  9. Data Construction
  10. Experiments
  11. Experimental Setup
  12. Training Data
  13. Training Details
  14. Evaluation Metrics
  15. Baseline Models
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Inference pipeline of WhispEar for bidirectional conversion.
  • Figure 2: Training pipeline of WhispEar.
  • Figure 3: Scaling experiment with pseudo-parallel data. Blue curves denote models trained with pseudo-data pretraining only, while red curves denote models further fine-tuned on aligned real data after pretraining.