Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like

Naoyuki Kanda, Xiaofei Wang, Sefik Emre Eskimez, Manthan Thakker, Hemin Yang, Zirun Zhu, Min Tang, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Yufei Xia, Jinzhu Li, Yanqing Liu, Sheng Zhao, Michael Zeng

TL;DR

This work introduces ELaTE, a zero-shot TTS that can generate natural laughter for any speaker with precise control over when and how laughter occurs. Built on conditional flow-matching, ELaTE incorporates frame-level laughter conditioning derived from a laughter detector and learns through a data-mixing fine-tuning scheme that combines small laughter-conditioned data with large-scale pretraining data. Evaluations on LibriSpeech and the DiariST-AliMeeting laughter test set show that ELaTE yields higher quality, more controllable laughing speech than prior zero-shot TTS and between baselines and S2ST models, without sacrificing intelligibility. The approach enables targeted, expressive adjustments in TTS and has practical implications for conversational agents and speech-to-speech translation where nuanced laughter transfer matters.

Abstract

Laughter is one of the most expressive and natural aspects of human speech, conveying emotions, social cues, and humor. However, most text-to-speech (TTS) systems lack the ability to produce realistic and appropriate laughter sounds, limiting their applications and user experience. While there have been prior works to generate natural laughter, they fell short in terms of controlling the timing and variety of the laughter to be generated. In this work, we propose ELaTE, a zero-shot TTS that can generate natural laughing speech of any speaker based on a short audio prompt with precise control of laughter timing and expression. Specifically, ELaTE works on the audio prompt to mimic the voice characteristic, the text prompt to indicate the contents of the generated speech, and the input to control the laughter expression, which can be either the start and end times of laughter, or the additional audio prompt that contains laughter to be mimicked. We develop our model based on the foundation of conditional flow-matching-based zero-shot TTS, and fine-tune it with frame-level representation from a laughter detector as additional conditioning. With a simple scheme to mix small-scale laughter-conditioned data with large-scale pre-training data, we demonstrate that a pre-trained zero-shot TTS model can be readily fine-tuned to generate natural laughter with precise controllability, without losing any quality of the pre-trained zero-shot TTS model. Through objective and subjective evaluations, we show that ELaTE can generate laughing speech with significantly higher quality and controllability compared to conventional models. See https://aka.ms/elate/ for demo samples.

Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like

TL;DR

This work introduces ELaTE, a zero-shot TTS that can generate natural laughter for any speaker with precise control over when and how laughter occurs. Built on conditional flow-matching, ELaTE incorporates frame-level laughter conditioning derived from a laughter detector and learns through a data-mixing fine-tuning scheme that combines small laughter-conditioned data with large-scale pretraining data. Evaluations on LibriSpeech and the DiariST-AliMeeting laughter test set show that ELaTE yields higher quality, more controllable laughing speech than prior zero-shot TTS and between baselines and S2ST models, without sacrificing intelligibility. The approach enables targeted, expressive adjustments in TTS and has practical implications for conversational agents and speech-to-speech translation where nuanced laughter transfer matters.

Abstract

Laughter is one of the most expressive and natural aspects of human speech, conveying emotions, social cues, and humor. However, most text-to-speech (TTS) systems lack the ability to produce realistic and appropriate laughter sounds, limiting their applications and user experience. While there have been prior works to generate natural laughter, they fell short in terms of controlling the timing and variety of the laughter to be generated. In this work, we propose ELaTE, a zero-shot TTS that can generate natural laughing speech of any speaker based on a short audio prompt with precise control of laughter timing and expression. Specifically, ELaTE works on the audio prompt to mimic the voice characteristic, the text prompt to indicate the contents of the generated speech, and the input to control the laughter expression, which can be either the start and end times of laughter, or the additional audio prompt that contains laughter to be mimicked. We develop our model based on the foundation of conditional flow-matching-based zero-shot TTS, and fine-tune it with frame-level representation from a laughter detector as additional conditioning. With a simple scheme to mix small-scale laughter-conditioned data with large-scale pre-training data, we demonstrate that a pre-trained zero-shot TTS model can be readily fine-tuned to generate natural laughter with precise controllability, without losing any quality of the pre-trained zero-shot TTS model. Through objective and subjective evaluations, we show that ELaTE can generate laughing speech with significantly higher quality and controllability compared to conventional models. See https://aka.ms/elate/ for demo samples.
Paper Structure (21 sections, 1 equation, 5 figures, 3 tables)

This paper contains 21 sections, 1 equation, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Zero-shot TTS
  4. Conditioning generative models
  5. Method
  6. Overview
  7. Conditional flow-matching
  8. Model architecture
  9. Fine-tuning pre-trained zero-shot TTS with additional conditioning
  10. Experiments
  11. Training data
  12. Model and training configurations
  13. Evaluation data
  14. Objective evaluation metrics
  15. Subjective evaluation metrics
  16. ...and 6 more sections

Figures (5)

  • Figure 1: An overview of the capability of ELaTE. ELaTE can generate natural laughing speech from a speaker prompt to mimic the voice characteristic, a text prompt to indicate the contents of the generated speech, and an additional input to control the laughter expression. The laughter, including the choice not to laugh, can be controlled either by (a) specifying the start and end times for laughing or (b) by using an audio example that contains laughter to be mimicked. (c) ELaTE is particularly beneficial for speech-to-speech translation that precisely transfers the nuance in the source speech. This is achieved by combining it with speech-to-text translation (S2TT).
  • Figure 2: An overview of (a) training and (b) inference of the flow-matching-based zero-shot TTS with laughter expression control.
  • Figure 3: An example of (a) source Chinese audio, (b) translated speech with our baseline TTS, and (c) translated speech with ELaTE.
  • Figure 4: Generated speech by ELaTE with speaker probability where the laughter probability is set to be (a) 0 for all frames, (b) 1 for the first 1.4 seconds and 0 for the rest, and (c) 0 for the first 1.4 seconds and 1 for the rest. The speaker prompt was taken from the speaker 1089 of LibriSpeech test-clean, and the text prompt was set to "that’s funny". In this example, we added 1 second of silence frames at the start and end of the estimated duration.
  • Figure 5: Subjective evaluation results of 30 samples from the DiariST-AliMeeting laughter test set, along with 95% confidence interval. Naturalness, speaker similarity, and laughter similarity were assessed.