EDNet: A Versatile Speech Enhancement Framework with Gating Mamba Mechanism and Phase Shift-Invariant Training

TL;DR

EDNet introduces two key innovations for versatile speech enhancement: a Gating Mamba (GM) module that adaptively blends masking (Erase) and mapping (Draw) on a region-wise basis, and Phase Shift-Invariant Training (PSIT) that relaxes strict phase alignment during training. The dual-stream architecture processes magnitude and phase separately, with PSIT applied during training to improve phase estimation without changing inference. Across denoising, dereverberation, bandwidth extension, and multi-distortion tasks, EDNet achieves state-of-the-art or competitive results, with PSIT providing robust gains in phase reconstruction and training stability. The work demonstrates that architectural adaptability combined with shift-tolerant supervision yields a practical, task-agnostic framework capable of handling complex real-world distortions while maintaining perceptual quality and content fidelity.

Abstract

Speech signals in real-world environments are frequently affected by various distortions such as additive noise, reverberation, and bandwidth limitation, which may appear individually or in combination. Traditional speech enhancement methods typically rely on either masking, which focuses on suppressing non-speech components while preserving observable structure, or mapping, which seeks to recover clean speech through direct transformation of the input. Each approach offers strengths in specific scenarios but may be less effective outside its target conditions. We propose the Erase and Draw Network (EDNet), a versatile speech enhancement framework designed to handle a broad range of distortion types without prior assumptions about task or input characteristics. EDNet consists of two main components: (1) the Gating Mamba (GM) module, which adaptively combines masking and mapping through a learnable gating mechanism that selects between suppression (Erase) and reconstruction (Draw) based on local signal features, and (2) Phase Shift-Invariant Training (PSIT), a shift tolerant supervision strategy that improves phase estimation by enabling dynamic alignment during training while remaining compatible with standard loss functions. Experimental results on denoising, dereverberation, bandwidth extension, and multi distortion enhancement tasks show that EDNet consistently achieves strong performance across conditions, demonstrating its architectural flexibility and adaptability to diverse task settings.

Figures (8)

• Figure 1: Overview of EDNet architecture. The model employs separate magnitude and phase enhancement streams to handle their distinct characteristics. PSIT is applied only during training to improve phase reconstruction without affecting the inference pipeline.
• Figure 2: Detailed architecture of the GM block. $h_N$ represents the hidden feature from the previous block, while $h_0$ refer to the initial hidden feature from the encoder.
• Figure 3: Visualization of gate maps in task-specific scenarios using single-distortion models. (a) Clean source speech. Gate maps of (b) the denoising model given additive noise input, (c) the dereverberation model given reverberant input, and (d) the bandwidth extension model given 8 kHz bandwidth-limited input.
• Figure 4: Visualization of gate maps in input-specific scenarios using the multi-distortion model. Gate maps and output magnitude spectrogram of the multi-distortion model given input of (a) additive noise, (b) reverberation, (c) 8 kHz bandwidth limitation, and (d) all three distortions combined.
• Figure 5: Empirical efficiency comparison among CMGAN, SEMamba, MP-SENet and EDNet. (a) Real-time factor (RTF) and (b) GPU memory usage with increasing input duration.