Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesis

Dong Yang, Yiyi Cai, Yuki Saito, Lixu Wang, Hiroshi Saruwatari

TL;DR

Shallow Flow Matching (SFM) addresses inefficiencies in coarse-to-fine FM-based TTS by constructing intermediate states along CondOT paths derived from coarse representations and initiating generation from these states. The method uses an adaptive orthogonal projection during training to locate intermediate states and employs a single-segment piecewise flow to focus inference on the latter FM stages; an SFM strength parameter $\alpha$ further guides inference. Across multiple FM-backed TTS models (e.g., Matcha-TTS, CosyVoice, StableTTS), SFM yields consistent naturalness gains in objective and subjective evaluations and markedly accelerates inference with adaptive-step ODE solvers. The approach requires only lightweight SFM heads and minimal backbone changes, offering a practical and generalizable enhancement to flow-based generative modeling in TTS and potentially other diffusion/flow-driven tasks.

Abstract

We propose Shallow Flow Matching (SFM), a novel mechanism that enhances flow matching (FM)-based text-to-speech (TTS) models within a coarse-to-fine generation paradigm. Unlike conventional FM modules, which use the coarse representations from the weak generator as conditions, SFM constructs intermediate states along the FM paths from these representations. During training, we introduce an orthogonal projection method to adaptively determine the temporal position of these states, and apply a principled construction strategy based on a single-segment piecewise flow. The SFM inference starts from the intermediate state rather than pure noise, thereby focusing computation on the latter stages of the FM paths. We integrate SFM into multiple TTS models with a lightweight SFM head. Experiments demonstrate that SFM yields consistent gains in speech naturalness across both objective and subjective evaluations, and significantly accelerates inference when using adaptive-step ODE solvers. Demo and codes are available at https://ydqmkkx.github.io/SFMDemo/.

Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesis

TL;DR

Shallow Flow Matching (SFM) addresses inefficiencies in coarse-to-fine FM-based TTS by constructing intermediate states along CondOT paths derived from coarse representations and initiating generation from these states. The method uses an adaptive orthogonal projection during training to locate intermediate states and employs a single-segment piecewise flow to focus inference on the latter FM stages; an SFM strength parameter further guides inference. Across multiple FM-backed TTS models (e.g., Matcha-TTS, CosyVoice, StableTTS), SFM yields consistent naturalness gains in objective and subjective evaluations and markedly accelerates inference with adaptive-step ODE solvers. The approach requires only lightweight SFM heads and minimal backbone changes, offering a practical and generalizable enhancement to flow-based generative modeling in TTS and potentially other diffusion/flow-driven tasks.

Abstract

We propose Shallow Flow Matching (SFM), a novel mechanism that enhances flow matching (FM)-based text-to-speech (TTS) models within a coarse-to-fine generation paradigm. Unlike conventional FM modules, which use the coarse representations from the weak generator as conditions, SFM constructs intermediate states along the FM paths from these representations. During training, we introduce an orthogonal projection method to adaptively determine the temporal position of these states, and apply a principled construction strategy based on a single-segment piecewise flow. The SFM inference starts from the intermediate state rather than pure noise, thereby focusing computation on the latter stages of the FM paths. We integrate SFM into multiple TTS models with a lightweight SFM head. Experiments demonstrate that SFM yields consistent gains in speech naturalness across both objective and subjective evaluations, and significantly accelerates inference when using adaptive-step ODE solvers. Demo and codes are available at https://ydqmkkx.github.io/SFMDemo/.
Paper Structure (33 sections, 4 theorems, 41 equations, 2 figures, 11 tables, 2 algorithms)

This paper contains 33 sections, 4 theorems, 41 equations, 2 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Flow matching
  4. Classifier-free guidance
  5. Flow matching-based TTS models
  6. Method
  7. Theorems
  8. Proposed methods on coarse-to-fine FM-based TTS
  9. Orthogonal projection onto CondOT paths
  10. Single-segment piecewise flow
  11. Inference with SFM strength
  12. Experimental setup
  13. Datasets
  14. Model implementations
  15. Training and inference configurations
  16. ...and 18 more sections

Key Result

Theorem 1

For any random variable $\boldsymbol{x}_m \sim \mathcal{N}(t_m \boldsymbol{x}_1, \sigma^2_m\boldsymbol{I})$, where $t_m \in [0,\infty)$ and $\sigma_m \in (0,\infty)$, we define a transformation that maps $\boldsymbol{x}_m$ onto the conditional OT (CondOT) paths. The output distribution varies contin where $\boldsymbol{x}_0 \sim \mathcal{N}(\boldsymbol{0}, \boldsymbol{I})$, with corresponding $\tau

Figures (2)

  • Figure 1: Inference process. Left: standard FM; Right: proposed SFM.
  • Figure 2:

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2
  • Theorem 1
  • Theorem 2