Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Normalize Everything: A Preconditioned Magnitude-Preserving Architecture for Diffusion-Based Speech Enhancement

Julius Richter, Danilo de Oliveira, Timo Gerkmann

TL;DR

The paper tackles diffusion-based speech enhancement using a Schrödinger bridge formulation and introduces NESE, a preconditioned, magnitude-preserving ADM architecture. It develops time-dependent input/output preconditioning and two skip-scaling modes, enabling the model to target either environmental noise or clean speech, and adds input-conditioned features with learnable fusion. A post-training EMA analysis reveals that, unlike image-generation tasks, shorter EMA lengths improve speech metrics, offering practical guidance for EMA use in speech restoration. Empirically, NESE is competitive with strong baselines and demonstrates robustness in mismatched settings on VoiceBank-DEMAND and EARS-WHAM, validating its design choices for stable training and effective conditioning in diffusion-based speech enhancement.

Abstract

This paper presents a new framework for diffusion-based speech enhancement. Our method employs a Schroedinger bridge to transform the noisy speech distribution into the clean speech distribution. To stabilize and improve training, we employ time-dependent scalings of the inputs and outputs of the network, known as preconditioning. We consider two skip connection configurations, which either include or omit the current process state in the denoiser's output, enabling the network to predict either environmental noise or clean speech. Each approach leads to improved performance on different speech enhancement metrics. To maintain stable magnitude levels and balance during training, we use a magnitude-preserving network architecture that normalizes all activations and network weights to unit length. Additionally, we propose learning the contribution of the noisy input within each network block for effective input conditioning. After training, we apply a method to approximate different exponential moving average (EMA) profiles and investigate their effects on the speech enhancement performance. In contrast to image generation tasks, where longer EMA lengths often enhance mode coverage, we observe that shorter EMA lengths consistently lead to better performance on standard speech enhancement metrics. Code, audio examples, and checkpoints are available online.

Normalize Everything: A Preconditioned Magnitude-Preserving Architecture for Diffusion-Based Speech Enhancement

TL;DR

The paper tackles diffusion-based speech enhancement using a Schrödinger bridge formulation and introduces NESE, a preconditioned, magnitude-preserving ADM architecture. It develops time-dependent input/output preconditioning and two skip-scaling modes, enabling the model to target either environmental noise or clean speech, and adds input-conditioned features with learnable fusion. A post-training EMA analysis reveals that, unlike image-generation tasks, shorter EMA lengths improve speech metrics, offering practical guidance for EMA use in speech restoration. Empirically, NESE is competitive with strong baselines and demonstrates robustness in mismatched settings on VoiceBank-DEMAND and EARS-WHAM, validating its design choices for stable training and effective conditioning in diffusion-based speech enhancement.

Abstract

This paper presents a new framework for diffusion-based speech enhancement. Our method employs a Schroedinger bridge to transform the noisy speech distribution into the clean speech distribution. To stabilize and improve training, we employ time-dependent scalings of the inputs and outputs of the network, known as preconditioning. We consider two skip connection configurations, which either include or omit the current process state in the denoiser's output, enabling the network to predict either environmental noise or clean speech. Each approach leads to improved performance on different speech enhancement metrics. To maintain stable magnitude levels and balance during training, we use a magnitude-preserving network architecture that normalizes all activations and network weights to unit length. Additionally, we propose learning the contribution of the noisy input within each network block for effective input conditioning. After training, we apply a method to approximate different exponential moving average (EMA) profiles and investigate their effects on the speech enhancement performance. In contrast to image generation tasks, where longer EMA lengths often enhance mode coverage, we observe that shorter EMA lengths consistently lead to better performance on standard speech enhancement metrics. Code, audio examples, and checkpoints are available online.
Paper Structure (15 sections, 29 equations, 1 figure, 3 tables)

This paper contains 15 sections, 29 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Schrödinger Bridge for Speech Enhancement
  4. Magnitude-Preserving Learned Layers
  5. Post-training Exponential Moving Average (EMA)
  6. Method
  7. Preconditioning
  8. Input Scaling
  9. Output Scaling
  10. Skip Scaling
  11. Loss Weighting
  12. Network architecture
  13. Experimental Setup
  14. Results
  15. Conclusion

Figures (1)

  • Figure 1: Speech enhancement performance as a function of EMA length on the validation set of VoiceBank-DEMAND.