AMDM-SE: Attention-based Multichannel Diffusion Model for Speech Enhancement
Renana Opochinsky, Sharon Gannot
TL;DR
This paper introduces AMDM-SE, a diffusion-based framework for multichannel speech enhancement that explicitly exploits inter-channel spatial cues through a cross-channel time-frequency attention mechanism. Building on the SGMSE backbone, AMDM-SE conditions the diffusion process on all microphone channels and uses a cross-channel TF-Attention module to generate multiple channel-conditioned representations, enabling faithful reconstruction of fine-grained speech details in the complex STFT domain. Evaluations on simulated data and the CHiME-3 dataset show that AMDM-SE consistently improves perceptual quality and intelligibility over single-channel diffusion and multichannel baselines without attention, highlighting the value of incorporating targeted multichannel spatial modeling into generative speech enhancement. The work suggests that diffusion models combined with multichannel spatial priors offer a promising direction for robust noise reduction in real-world microphone arrays, with potential extensions to incorporate additional multichannel features such as relative transfer function information.
Abstract
Diffusion models have recently achieved impressive results in reconstructing images from noisy inputs, and similar ideas have been applied to speech enhancement by treating time-frequency representations as images. With the ubiquity of multi-microphone devices, we extend state-of-the-art diffusion-based methods to exploit multichannel inputs for improved performance. Multichannel diffusion-based enhancement remains in its infancy, with prior work making limited use of advanced mechanisms such as attention for spatial modeling - a gap addressed in this paper. We propose AMDM-SE, an Attention-based Multichannel Diffusion Model for Speech Enhancement, designed specifically for noise reduction. AMDM-SE leverages spatial inter-channel information through a novel cross-channel time-frequency attention block, enabling faithful reconstruction of fine-grained signal details within a generative diffusion framework. On the CHiME-3 benchmark, AMDM-SE outperforms both a single-channel diffusion baseline and a multichannel model without attention, as well as a strong DNN-based predictive method. Simulated-data experiments further underscore the importance of the proposed multichannel attention mechanism. Overall, our results show that incorporating targeted multichannel attention into diffusion models substantially improves noise reduction. While multichannel diffusion-based speech enhancement is still an emerging field, our work contributes a new and complementary approach to the growing body of research in this direction.