BSS-CFFMA: Cross-Domain Feature Fusion and Multi-Attention Speech Enhancement Network based on Self-Supervised Embedding

TL;DR

The paper tackles the limited use of self-supervised learning (SSL) features in single-channel speech enhancement (SE) and introduces BSS-CFFMA, which fuses SSL embeddings with spectrogram features via a multi-scale cross-domain feature fusion (MSCFF) module and refines representations with a residual hybrid multi-attention (RHMA) block. It employs a weighted-sum SSL representation $F_{ws:ssl} = \sum_{i=0}^{N-1} e(i) z(i)$ with $0 \le e(i) \le 1$ and $\sum_i e(i)=1$, integrates this with spectral features through MSCFF, and enhances the fused features using MHSA, FFN, and the SCTA-based fusion. Results on VoiceBank-DEMAND demonstrate state-of-the-art performance, with additional evaluation on WHAMR! showing strong denoising, dereverberation, and joint capabilities, underscoring the effectiveness of SSL embedding-based cross-domain fusion for complex SE tasks. The work identifies phase processing as a bottleneck and points to future directions in phase optimization to pushSE performance further, with an open-source demo available online.

Abstract

Speech self-supervised learning (SSL) represents has achieved state-of-the-art (SOTA) performance in multiple downstream tasks. However, its application in speech enhancement (SE) tasks remains immature, offering opportunities for improvement. In this study, we introduce a novel cross-domain feature fusion and multi-attention speech enhancement network, termed BSS-CFFMA, which leverages self-supervised embeddings. BSS-CFFMA comprises a multi-scale cross-domain feature fusion (MSCFF) block and a residual hybrid multi-attention (RHMA) block. The MSCFF block effectively integrates cross-domain features, facilitating the extraction of rich acoustic information. The RHMA block, serving as the primary enhancement module, utilizes three distinct attention modules to capture diverse attention representations and estimate high-quality speech signals. We evaluate the performance of the BSS-CFFMA model through comparative and ablation studies on the VoiceBank-DEMAND dataset, achieving SOTA results. Furthermore, we select three types of data from the WHAMR! dataset, a collection specifically designed for speech enhancement tasks, to assess the capabilities of BSS-CFFMA in tasks such as denoising only, dereverberation only, and simultaneous denoising and dereverberation. This study marks the first attempt to explore the effectiveness of self-supervised embedding-based speech enhancement methods in complex tasks encompassing dereverberation and simultaneous denoising and dereverberation. The demo implementation of BSS-CFFMA is available online\footnote[2]{https://github.com/AlimMat/BSS-CFFMA. \label{s1}}.

Figures (7)

• Figure 1: Architecture of the proposed BSS-CFFMA. "STFT" represents the short-time Fourier transform of the speech. "Spec" stands for Spectrogram. $\downarrow$ represents the downsampling operation which Liner and Z $\in \mathbb{R}^{T*512}$. $\Downarrow$ represents the downsampling operation which Liner and Sigmod. "iSTFT" represents the inverse short-time Fourier transform of the speech. "Reconstruct" represents the reconstruction of the speech complex spectrum for the speech spectrogram $\in \mathbb{R}^{F*T}$ and phase $\in \mathbb{R}^{F*T*2}$.
• Figure 2: Structure multi-scale cross-domain feature fusion (MSCFF) model. "DConv1d" represents the dilation convolution, where the kernel sizes are 3 and 5, respectively, and the dilation coefficients are 1.
• Figure 3: Structure of selective channel-attention fusion (SCA) block. Where FC consists of two Liner and one Relu activation function.
• Figure 4: Structure of selective time-attention fusion (STA) block. Where the Convolution kernel sizes are 3 and 5, respectively.
• Figure 5: Pairwise comparison of PESQ with BSS-CFFMA with baseline (BSS-SE) on VoiceBank-DEMAND nad WHAMR! dataset test. Where SSL uses Wav2vec2.0 (without fine-tuning).