Diffusion Buffer for Online Generative Speech Enhancement

TL;DR

This work confirms that the online Diffusion Buffer also outperforms its predictive counterpart on unseen noisy speech data and designs a 2D convolutional UNet architecture that specifically aligns with the Diffusion Buffer's look-ahead.

Abstract

Online Speech Enhancement was mainly reserved for predictive models. A key advantage of these models is that for an incoming signal frame from a stream of data, the model is called only once for enhancement. In contrast, generative Speech Enhancement models often require multiple calls, resulting in a computational complexity that is too high for many online speech enhancement applications. This work presents the Diffusion Buffer, a generative diffusion-based Speech Enhancement model which only requires one neural network call per incoming signal frame from a stream of data and performs enhancement in an online fashion on a consumer-grade GPU. The key idea of the Diffusion Buffer is to align physical time with Diffusion time-steps. The approach progressively denoises frames through physical time, where past frames have more noise removed. Consequently, an enhanced frame is output to the listener with a delay defined by the Diffusion Buffer, and the output frame has a corresponding look-ahead. In this work, we extend upon our previous work by carefully designing a 2D convolutional UNet architecture that specifically aligns with the Diffusion Buffer's look-ahead. We observe that the proposed UNet improves performance, particularly when the algorithmic latency is low. Moreover, we show that using a Data Prediction loss instead of Denoising Score Matching loss enables flexible control over the trade-off between algorithmic latency and quality during inference. The extended Diffusion Buffer equipped with a novel NN and loss function drastically reduces the algorithmic latency from 320 - 960 ms to 32 - 176 ms with an even increased performance. While it has been shown before that offline generative diffusion models outperform predictive approaches in unseen noisy speech data, we confirm that the online Diffusion Buffer also outperforms its predictive counterpart on unseen noisy speech data.

Figures (5)

• Figure 1: Diffusion Buffer illustration. Input to the NN is the noisy chunk $Y_c$ and perturbed input $\mathbf V^{\overrightarrow{t}}$, which contains the Diffusion Buffer. Frames within the Diffusion Buffer are modeled at different Diffusion-timesteps (white numbers in frame). In this example, $Y_c$ and $\mathbf V^{\overrightarrow{t}}$ contain $K=5$ frames, and the Diffusion Buffer contains $B=4$ frames.
• Figure 2: Two examples of receptive fields for an input sequence of length 43. The global stride of the network is 8. (a) Applying the three proposed modifications to the convolutional layers. The receptive field has block-causal dependencies. The look-ahead is at most equal to the global stride $g$. (b) Default down- and upsampling operations. The receptive is symmetrically placed around blocks and has a large look-ahead.
• Figure 3: Proposed downsampling and upsampling on an input sequence $X$. Blue tokens represent zero-padding. The red token in the output will be cropped out, it will not be part of the final output.
• Figure 4: Comparison of sncsnpp (symmetric receptive field) versus BC-NCSN++ (block-causal receptive field) on EARS-WHAM-v2. Chunk-based (online) processing performance plotted against different dint $d$. We see that sncsnpp degrades much more for small $d$ opposed to BC-NCSN++.
• Figure 5: Comparison between generative DB-BBED and its predictive counterpart on the mismatched EARS-General test set. Unlike the matched case in \ref{['tab:db_ablation']}, we can now see that the generative method outperforms the predictive method.