Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lombard Speech Synthesis for Any Voice with Controllable Style Embeddings

Seymanur Akti, Alexander Waibel

TL;DR

The paper tackles zero-shot Lombard speech synthesis for arbitrary speakers without Lombard-specific training data by learning prosodic style embeddings from a large dataset and linking their variance to Lombard attributes via PCA. It integrates these insights into an enhanced F5-TTS system (F5-TTS Style) using a fixed-size style embedding, FiLM conditioning, and an ECAPA-TDNN encoder, with controlled duration to enable explicit Lombard control. The method enables cross-lingual adaptation and preserves speaker identity while improving intelligibility in noisy environments, as demonstrated by WER, SSIM, and UTMOS evaluations and robust noise tests. This approach is scalable and practical for hearing-assistive applications and robust TTS in challenging conditions, offering a flexible and interpretable pathway to controllable Lombard synthesis.

Abstract

The Lombard effect plays a key role in natural communication, particularly in noisy environments or when addressing hearing-impaired listeners. We present a controllable text-to-speech (TTS) system capable of synthesizing Lombard speech for any speaker without requiring explicit Lombard data during training. Our approach leverages style embeddings learned from a large, prosodically diverse dataset and analyzes their correlation with Lombard attributes using principal component analysis (PCA). By shifting the relevant PCA components, we manipulate the style embeddings and incorporate them into our TTS model to generate speech at desired Lombard levels. Evaluations demonstrate that our method preserves naturalness and speaker identity, enhances intelligibility under noise, and provides fine-grained control over prosody, offering a robust solution for controllable Lombard TTS for any speaker.

Lombard Speech Synthesis for Any Voice with Controllable Style Embeddings

TL;DR

The paper tackles zero-shot Lombard speech synthesis for arbitrary speakers without Lombard-specific training data by learning prosodic style embeddings from a large dataset and linking their variance to Lombard attributes via PCA. It integrates these insights into an enhanced F5-TTS system (F5-TTS Style) using a fixed-size style embedding, FiLM conditioning, and an ECAPA-TDNN encoder, with controlled duration to enable explicit Lombard control. The method enables cross-lingual adaptation and preserves speaker identity while improving intelligibility in noisy environments, as demonstrated by WER, SSIM, and UTMOS evaluations and robust noise tests. This approach is scalable and practical for hearing-assistive applications and robust TTS in challenging conditions, offering a flexible and interpretable pathway to controllable Lombard synthesis.

Abstract

The Lombard effect plays a key role in natural communication, particularly in noisy environments or when addressing hearing-impaired listeners. We present a controllable text-to-speech (TTS) system capable of synthesizing Lombard speech for any speaker without requiring explicit Lombard data during training. Our approach leverages style embeddings learned from a large, prosodically diverse dataset and analyzes their correlation with Lombard attributes using principal component analysis (PCA). By shifting the relevant PCA components, we manipulate the style embeddings and incorporate them into our TTS model to generate speech at desired Lombard levels. Evaluations demonstrate that our method preserves naturalness and speaker identity, enhances intelligibility under noise, and provides fine-grained control over prosody, offering a robust solution for controllable Lombard TTS for any speaker.
Paper Structure (10 sections, 1 equation, 3 figures, 5 tables)

This paper contains 10 sections, 1 equation, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Model Adaptation
  4. Lombardness Control
  5. EXPERIMENTS AND RESULTS
  6. Comparison with Baseline
  7. Intelligibility Robustness of Lombard Speech
  8. Speaker Identity Preservation in Lombard Speech
  9. CONCLUSION
  10. ACKNOWLEDGEMENTS

Figures (3)

  • Figure 1: Training (left) and inference (right) pipelines of the proposed system. Blue blocks represent components inherited from the original F5-TTS model, while green blocks indicate modules introduced in the proposed version.
  • Figure 2: PCA analysis of style embeddings.
  • Figure 3: WER and $\Delta$WER trends for GT and TTS.