Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-Time Streamable Generative Speech Restoration with Flow Matching

Simon Welker, Bunlong Lay, Maris Hillemann, Tal Peer, Timo Gerkmann

TL;DR

Stream.FM introduces a real-time, streaming-capable generative speech restoration framework based on frame-causal flow matching. By combining buffered multi-step inference, a tailored frame-causal DNN, predictive-generative enhancement, and learned low-NFE ODE solvers, the method achieves sub-50 ms total latency on consumer GPUs while handling diverse tasks such as enhancement, dereverberation, bandwidth extension, and vocoding. The work demonstrates state-of-the-art performance among streaming methods through objective metrics and listening tests, and shows that careful solver design and model compression can further improve the compute/quality tradeoff. The proposed approach is accompanied by a public codebase and supplementary materials that facilitate reproducibility and further research in real-time generative speech restoration.

Abstract

Diffusion-based generative models have greatly impacted the speech processing field in recent years, exhibiting high speech naturalness and spawning a new research direction. Their application in real-time communication is, however, still lagging behind due to their computation-heavy nature involving multiple calls of large DNNs. Here, we present Stream.FM, a frame-causal flow-based generative model with an algorithmic latency of 32 milliseconds (ms) and a total latency of 48 ms, paving the way for generative speech processing in real-time communication. We propose a buffered streaming inference scheme and an optimized DNN architecture, show how learned few-step numerical solvers can boost output quality at a fixed compute budget, explore model weight compression to find favorable points along a compute/quality tradeoff, and contribute a model variant with 24 ms total latency for the speech enhancement task. Our work looks beyond theoretical latencies, showing that high-quality streaming generative speech processing can be realized on consumer GPUs available today. Stream.FM can solve a variety of speech processing tasks in a streaming fashion: speech enhancement, dereverberation, codec post-filtering, bandwidth extension, STFT phase retrieval, and Mel vocoding. As we verify through comprehensive evaluations and a MUSHRA listening test, Stream.FM establishes a state-of-the-art for generative streaming speech restoration, exhibits only a reasonable reduction in quality compared to a non-streaming variant, and outperforms our recent work (Diffusion Buffer) on generative streaming speech enhancement while operating at a lower latency.

Real-Time Streamable Generative Speech Restoration with Flow Matching

TL;DR

Stream.FM introduces a real-time, streaming-capable generative speech restoration framework based on frame-causal flow matching. By combining buffered multi-step inference, a tailored frame-causal DNN, predictive-generative enhancement, and learned low-NFE ODE solvers, the method achieves sub-50 ms total latency on consumer GPUs while handling diverse tasks such as enhancement, dereverberation, bandwidth extension, and vocoding. The work demonstrates state-of-the-art performance among streaming methods through objective metrics and listening tests, and shows that careful solver design and model compression can further improve the compute/quality tradeoff. The proposed approach is accompanied by a public codebase and supplementary materials that facilitate reproducibility and further research in real-time generative speech restoration.

Abstract

Diffusion-based generative models have greatly impacted the speech processing field in recent years, exhibiting high speech naturalness and spawning a new research direction. Their application in real-time communication is, however, still lagging behind due to their computation-heavy nature involving multiple calls of large DNNs. Here, we present Stream.FM, a frame-causal flow-based generative model with an algorithmic latency of 32 milliseconds (ms) and a total latency of 48 ms, paving the way for generative speech processing in real-time communication. We propose a buffered streaming inference scheme and an optimized DNN architecture, show how learned few-step numerical solvers can boost output quality at a fixed compute budget, explore model weight compression to find favorable points along a compute/quality tradeoff, and contribute a model variant with 24 ms total latency for the speech enhancement task. Our work looks beyond theoretical latencies, showing that high-quality streaming generative speech processing can be realized on consumer GPUs available today. Stream.FM can solve a variety of speech processing tasks in a streaming fashion: speech enhancement, dereverberation, codec post-filtering, bandwidth extension, STFT phase retrieval, and Mel vocoding. As we verify through comprehensive evaluations and a MUSHRA listening test, Stream.FM establishes a state-of-the-art for generative streaming speech restoration, exhibits only a reasonable reduction in quality compared to a non-streaming variant, and outperforms our recent work (Diffusion Buffer) on generative streaming speech enhancement while operating at a lower latency.
Paper Structure (49 sections, 20 equations, 3 figures, 9 tables)

This paper contains 49 sections, 20 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Diffusion- and flow-based speech processing
  4. Latency definitions
  5. Explicit Runge-Kutta ODE solvers
  6. Related work
  7. Methods
  8. Multi-step streaming diffusion
  9. Efficient real-time streaming diffusion inference
  10. DNN architecture
  11. Predictive-generative speech enhancement
  12. Custom learned low-NFE ODE solvers
  13. Model compression through weight decoupling
  14. Experiments
  15. Speech restoration tasks
  16. ...and 34 more sections

Figures (3)

  • Figure 1: Inference for one new frame (orange) in a simplified frame-causal DNN. While the output frame has a receptive field size (yellow) of 9 in the input, only 3 frames must be evaluated in each layer since all required past results (green) can be stored in a buffer $\mathbf B$.
  • Figure 2: Violin plots of the scores listeners assigned to examples from each method in the listening experiments for (a) speech enhancement and (b) bandwidth extension. SFM is Stream.FM, FM is the flow matching baseline, and DB is the Diffusion Buffer lay2025diffusionjpre. Note that FM has $\ell_\text{alg}=\infty$ and DB with $d=9$ has $\ell_\text{alg}\approx180$ ms.
  • Figure 3: Metrics of compressed Stream.FM models for Mel vocoding using kernel ranks $J \in \{2,4,6,9\}$ where $J=9$ is uncompressed, using the Euler solver. We compare the maximum NFE for each $J$ under our runtime budget against constant $\text{NFE}=5$ and a high-NFE variant $K=9,\ \text{NFE}=25$. Reported GFLOPs are per-frame.