FLOWER: Flow-Based Estimated Gaussian Guidance for General Speech Restoration

TL;DR

FLOWER addresses general speech restoration by introducing Gaussian guidance derived from a conditional normalizing flow, integrated as conditioning within diffusion- and flow-matching-based generative models. The Gaussian guidance encodes oracle-like information from clean speech and is used during training, then sampled from a Gaussian at inference to enable efficient conditioning without the NF. The approach encompasses time-adaptive scaling and extends to flow-matching with an OT path to improve sampling efficiency, delivering superior restoration across noise, reverberation, and bandwidth extension on matched and mismatched datasets. Empirically, FLOWER improves PESQ, CSIG, CBAK, COVL, SRMR, and LSD while reducing the required number of sampling steps, demonstrating practical impact for real-world speech restoration tasks.

Abstract

We introduce FLOWER, a novel conditioning method designed for speech restoration that integrates Gaussian guidance into generative frameworks. By transforming clean speech into a predefined prior distribution (e.g., Gaussian distribution) using a normalizing flow network, FLOWER extracts critical information to guide generative models. This guidance is incorporated into each block of the generative network, enabling precise restoration control. Experimental results demonstrate the effectiveness of FLOWER in improving performance across various general speech restoration tasks.

Figures (6)

• Figure 1: Normalizing flow model architecture based on rational-quadratic transform. $x$, $c$, and $z$ are data (clean speech), a conditional feature (latent representation of diffusion model), and Gaussian noise, respectively. Conv block consists of dilated convolutions, and the number of block $N$ is 4.
• Figure 2: The overall architecture of FLOWER approach. Gaussian guidance is extracted from the NF network during the training (a), but it is extracted from a Gaussian distribution during the inference (b). In the training process, the latent feature $c$ passes through a Conv2D layer (256, 1, 1, 1), and the input speech spectrogram passes through three Conv2D layers, each (1, 1, 4, 4). Subsequently, the output $z$ of the NF network passes through a ConvTranspose 2D layer (1, 256, 32, 32) and (1, 128, 64, 64) respectively, before being added to the diffusion network. Each shape represents (input channels, output channels, kernel size, and stride).
• Figure 3: Comparison between the restored spectrograms on "matched" scenario.
• Figure 4: Performance comparison between the SGMSE+ model and the FLOWER method on SGMSE+ and FGMSE+ according to the number of sampling steps ($N$=15, 25) under the "matched" scenario, respectively. The $x$-axis represents the number of sampling steps ($N$), while the $y$-axis indicates the metric scores. Higher scores are better for PESQ, CBAK, and SRMR, while lower scores are better for LSD. Colors: SGMSE+ (blue), FLOWER on SGMSE+ (red), FLOWER on FGMSE+ (yellow), Time-adaptive FLOWER on FGMSE+ (green).
• Figure 5: Spectrogram analysis comparing the proposed method to baseline models. The results demonstrate the efficacy of the proposed approach in preserving spectral details and mitigating distortions. Notably, the "Time-adaptive FLOWER on FGMSE+" model further alleviated residual reverberations present in the original signals.