Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ELF: Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis

Jungil Kong, Junmo Lee, Jeongmin Kim, Beomjeong Kim, Jihoon Park, Dohee Kong, Changheon Lee, Sangjin Kim

TL;DR

ELF tackles the challenge of modeling many speakers without target-speaker training by learning discretized speaker-specific latent features through a variational autoencoder (SFEN) and conditioning a text-to-speech system via an attention-based fusion of a discrete codebook. The two-stage design enables accurate expression of overall speaker characteristics, supports zero-shot and artificial-speaker generation through blending, and demonstrates complete reconstruction of target speech from encoded features. Empirical results across LibriTTS and VCTK show ELF achieving high speaker similarity, robust intelligibility, and strong cross-lingual performance, often surpassing zero-shot and multi-speaker baselines. The work proposes a flexible methodology for encoding and reconstructing speaker traits applicable to diverse tasks, while highlighting potential misuse and the need for detection and watermarking solutions.

Abstract

In this work, we propose a novel method for modeling numerous speakers, which enables expressing the overall characteristics of speakers in detail like a trained multi-speaker model without additional training on the target speaker's dataset. Although various works with similar purposes have been actively studied, their performance has not yet reached that of trained multi-speaker models due to their fundamental limitations. To overcome previous limitations, we propose effective methods for feature learning and representing target speakers' speech characteristics by discretizing the features and conditioning them to a speech synthesis model. Our method obtained a significantly higher similarity mean opinion score (SMOS) in subjective similarity evaluation than seen speakers of a high-performance multi-speaker model, even with unseen speakers. The proposed method also outperforms a zero-shot method by significant margins. Furthermore, our method shows remarkable performance in generating new artificial speakers. In addition, we demonstrate that the encoded latent features are sufficiently informative to reconstruct an original speaker's speech completely. It implies that our method can be used as a general methodology to encode and reconstruct speakers' characteristics in various tasks.

ELF: Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis

TL;DR

ELF tackles the challenge of modeling many speakers without target-speaker training by learning discretized speaker-specific latent features through a variational autoencoder (SFEN) and conditioning a text-to-speech system via an attention-based fusion of a discrete codebook. The two-stage design enables accurate expression of overall speaker characteristics, supports zero-shot and artificial-speaker generation through blending, and demonstrates complete reconstruction of target speech from encoded features. Empirical results across LibriTTS and VCTK show ELF achieving high speaker similarity, robust intelligibility, and strong cross-lingual performance, often surpassing zero-shot and multi-speaker baselines. The work proposes a flexible methodology for encoding and reconstructing speaker traits applicable to diverse tasks, while highlighting potential misuse and the need for detection and watermarking solutions.

Abstract

In this work, we propose a novel method for modeling numerous speakers, which enables expressing the overall characteristics of speakers in detail like a trained multi-speaker model without additional training on the target speaker's dataset. Although various works with similar purposes have been actively studied, their performance has not yet reached that of trained multi-speaker models due to their fundamental limitations. To overcome previous limitations, we propose effective methods for feature learning and representing target speakers' speech characteristics by discretizing the features and conditioning them to a speech synthesis model. Our method obtained a significantly higher similarity mean opinion score (SMOS) in subjective similarity evaluation than seen speakers of a high-performance multi-speaker model, even with unseen speakers. The proposed method also outperforms a zero-shot method by significant margins. Furthermore, our method shows remarkable performance in generating new artificial speakers. In addition, we demonstrate that the encoded latent features are sufficiently informative to reconstruct an original speaker's speech completely. It implies that our method can be used as a general methodology to encode and reconstruct speakers' characteristics in various tasks.
Paper Structure (27 sections, 5 equations, 7 figures, 8 tables)

This paper contains 27 sections, 5 equations, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Overview
  4. Speech Feature Encoding
  5. Text-to-Speech with Speech Feature Condition
  6. Speaker Blending
  7. Speech Feature-to-Speech with Text Condition
  8. Experiments
  9. Datasets
  10. Speech Feature Encoding
  11. Text-to-Speech
  12. Speech Feature-to-Speech
  13. Results
  14. Text-to-Speech
  15. Speaker Blending
  16. ...and 12 more sections

Figures (7)

  • Figure 1: (a) Training procedure of SFEN. (b) Inference procedure of SFEN.
  • Figure 2: (a) Training procedure of TTS model. (b) Inference procedure of TTS model.
  • Figure 3: Prior encoder of speaker blending.
  • Figure 4: Visualization of two distributions. PCA is used for dimensionality reduction.
  • Figure 5: Visualization of the speaker vectors from synthesized speeches
  • ...and 2 more figures