Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesis

Zehua Chen, Guande He, Kaiwen Zheng, Xu Tan, Jun Zhu

TL;DR

Bridge-TTS replaces the traditional data-to-noise diffusion prior with a clean, deterministic text-latent prior through a tractable Schrodinger bridge, enabling data-to-data mel-spectrogram generation. The authors derive a fully tractable SB between paired data, propose training and sampling objectives (including bridge SDE/ODE), and demonstrate state-of-the-art synthesis quality and sampling efficiency on LJ-Speech, outperforming Grad-TTS and fast diffusion baselines across various NFEs. They also provide comprehensive ablations on priors, noise schedules, and samplers, and connect the sampling schemes to DDIM and posterior-sampling concepts. This work introduces a new baseline for TTS that leverages informative priors to achieve high-fidelity, fast sampling, with potential applicability to other data-to-data generative tasks.

Abstract

In text-to-speech (TTS) synthesis, diffusion models have achieved promising generation quality. However, because of the pre-defined data-to-noise diffusion process, their prior distribution is restricted to a noisy representation, which provides little information of the generation target. In this work, we present a novel TTS system, Bridge-TTS, making the first attempt to substitute the noisy Gaussian prior in established diffusion-based TTS methods with a clean and deterministic one, which provides strong structural information of the target. Specifically, we leverage the latent representation obtained from text input as our prior, and build a fully tractable Schrodinger bridge between it and the ground-truth mel-spectrogram, leading to a data-to-data process. Moreover, the tractability and flexibility of our formulation allow us to empirically study the design spaces such as noise schedules, as well as to develop stochastic and deterministic samplers. Experimental results on the LJ-Speech dataset illustrate the effectiveness of our method in terms of both synthesis quality and sampling efficiency, significantly outperforming our diffusion counterpart Grad-TTS in 50-step/1000-step synthesis and strong fast TTS models in few-step scenarios. Project page: https://bridge-tts.github.io/

Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesis

TL;DR

Bridge-TTS replaces the traditional data-to-noise diffusion prior with a clean, deterministic text-latent prior through a tractable Schrodinger bridge, enabling data-to-data mel-spectrogram generation. The authors derive a fully tractable SB between paired data, propose training and sampling objectives (including bridge SDE/ODE), and demonstrate state-of-the-art synthesis quality and sampling efficiency on LJ-Speech, outperforming Grad-TTS and fast diffusion baselines across various NFEs. They also provide comprehensive ablations on priors, noise schedules, and samplers, and connect the sampling schemes to DDIM and posterior-sampling concepts. This work introduces a new baseline for TTS that leverages informative priors to achieve high-fidelity, fast sampling, with potential applicability to other data-to-data generative tasks.

Abstract

In text-to-speech (TTS) synthesis, diffusion models have achieved promising generation quality. However, because of the pre-defined data-to-noise diffusion process, their prior distribution is restricted to a noisy representation, which provides little information of the generation target. In this work, we present a novel TTS system, Bridge-TTS, making the first attempt to substitute the noisy Gaussian prior in established diffusion-based TTS methods with a clean and deterministic one, which provides strong structural information of the target. Specifically, we leverage the latent representation obtained from text input as our prior, and build a fully tractable Schrodinger bridge between it and the ground-truth mel-spectrogram, leading to a data-to-data process. Moreover, the tractability and flexibility of our formulation allow us to empirically study the design spaces such as noise schedules, as well as to develop stochastic and deterministic samplers. Experimental results on the LJ-Speech dataset illustrate the effectiveness of our method in terms of both synthesis quality and sampling efficiency, significantly outperforming our diffusion counterpart Grad-TTS in 50-step/1000-step synthesis and strong fast TTS models in few-step scenarios. Project page: https://bridge-tts.github.io/
Paper Structure (48 sections, 3 theorems, 60 equations, 8 figures, 5 tables, 2 algorithms)

This paper contains 48 sections, 3 theorems, 60 equations, 8 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Diffusion Models
  4. Diffusion-Based TTS Systems
  5. Schrodinger Bridge
  6. Bridge-TTS
  7. Schrodinger Bridge between Paired Data
  8. Model Training
  9. Sampling Scheme
  10. Bridge SDE
  11. Bridge ODE
  12. Experiments
  13. Training Setup
  14. Data
  15. Model training
  16. ...and 33 more sections

Key Result

Proposition 3.1

Assume $\bm{f}=f(t)\bm{x}_t$, the analytical solution to Eqn. (eq:SB-PDE) when $p_{\text{data}}=\mathcal{N}(\bm{x}_0,\epsilon^2\bm{I})$ and $p_{\text{prior}}=\mathcal{N}(\bm{x}_1,e^{2\int_0^1f(\tau)\mathrm{d}\tau}\epsilon^2\bm{I})$ is where $t\in [0,1]$, and

Figures (8)

  • Figure 1: An overview of Bridge-TTS built on Schrodinger bridge.
  • Figure 2: We show a 4-step ODE generation result of Grad-TTS Grad-TTS and Bridge-TTS in the left figure, and a 2-step ODE sampling trajectory of Bridge-TTS in the right one. The ground-truth mel-spectrogram is shown for comparison.
  • Figure 3: The scaling factor and variance in Grad-TTS and Bridge-TTS.
  • Figure 4: The preference test between Bridge-TTS and diffusion-based TTS systems.
  • Figure 5: The mel-spectrogram of synthesized (NFE=1000) and ground-truth LJ001-0006 and LJ002-0029.
  • ...and 3 more figures

Theorems & Definitions (5)

  • Proposition 3.1: Tractable Schrodinger Bridge between Gaussian-Smoothed Paired Data with Reference SDE of Linear Drift, proof in Appendix \ref{['appendix:proof-tractable-sb']}
  • Proposition 3.2: Exact Solution and First-Order Discretization of Bridge SDE/ODE, proof in Appendix \ref{['appendix:proof-sb-sampling']}
  • proof : Proof of Proposition \ref{['prop:tractable-sb']}
  • proof : Proof of Proposition \ref{['prop:sampler']}
  • Corollary B.1: 1-step First-Order Bridge SDE/ODE Sampler Recovers Direct Data Prediction