Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Heterogeneous Space Fusion and Dual-Dimension Attention: A New Paradigm for Speech Enhancement

Tao Zheng, Liejun Wang, Yinfeng Yu

TL;DR

This work tackles single-channel speech enhancement under noisy conditions by introducing HFSDA, a framework that fuses self-supervised embeddings with STFT spectrogram features and employs a dual-dimension attention mechanism. The method combines an ODConv-based spectrogram branch with a self-supervised semantic branch, then refines features through a Dual-Dimension Attention (DDA) module built atop a refined Conformer. Key contributions include the ODConv integration for multi-dimensional feature extraction, the FA-based frequency-focused DDA, and comprehensive ablations demonstrating the value of heterogeneous spatial feature fusion. Empirical results on the VCTK-DEMAND dataset show competitive performance across standard metrics (PESQ, STOI, etc.), indicating the practical impact of combining SSL representations with spectro-temporal processing for speech enhancement.

Abstract

Self-supervised learning has demonstrated impressive performance in speech tasks, yet there remains ample opportunity for advancement in the realm of speech enhancement research. In addressing speech tasks, confining the attention mechanism solely to the temporal dimension poses limitations in effectively focusing on critical speech features. Considering the aforementioned issues, our study introduces a novel speech enhancement framework, HFSDA, which skillfully integrates heterogeneous spatial features and incorporates a dual-dimension attention mechanism to significantly enhance speech clarity and quality in noisy environments. By leveraging self-supervised learning embeddings in tandem with Short-Time Fourier Transform (STFT) spectrogram features, our model excels at capturing both high-level semantic information and detailed spectral data, enabling a more thorough analysis and refinement of speech signals. Furthermore, we employ the innovative Omni-dimensional Dynamic Convolution (ODConv) technology within the spectrogram input branch, enabling enhanced extraction and integration of crucial information across multiple dimensions. Additionally, we refine the Conformer model by enhancing its feature extraction capabilities not only in the temporal dimension but also across the spectral domain. Extensive experiments on the VCTK-DEMAND dataset show that HFSDA is comparable to existing state-of-the-art models, confirming the validity of our approach.

Heterogeneous Space Fusion and Dual-Dimension Attention: A New Paradigm for Speech Enhancement

TL;DR

This work tackles single-channel speech enhancement under noisy conditions by introducing HFSDA, a framework that fuses self-supervised embeddings with STFT spectrogram features and employs a dual-dimension attention mechanism. The method combines an ODConv-based spectrogram branch with a self-supervised semantic branch, then refines features through a Dual-Dimension Attention (DDA) module built atop a refined Conformer. Key contributions include the ODConv integration for multi-dimensional feature extraction, the FA-based frequency-focused DDA, and comprehensive ablations demonstrating the value of heterogeneous spatial feature fusion. Empirical results on the VCTK-DEMAND dataset show competitive performance across standard metrics (PESQ, STOI, etc.), indicating the practical impact of combining SSL representations with spectro-temporal processing for speech enhancement.

Abstract

Self-supervised learning has demonstrated impressive performance in speech tasks, yet there remains ample opportunity for advancement in the realm of speech enhancement research. In addressing speech tasks, confining the attention mechanism solely to the temporal dimension poses limitations in effectively focusing on critical speech features. Considering the aforementioned issues, our study introduces a novel speech enhancement framework, HFSDA, which skillfully integrates heterogeneous spatial features and incorporates a dual-dimension attention mechanism to significantly enhance speech clarity and quality in noisy environments. By leveraging self-supervised learning embeddings in tandem with Short-Time Fourier Transform (STFT) spectrogram features, our model excels at capturing both high-level semantic information and detailed spectral data, enabling a more thorough analysis and refinement of speech signals. Furthermore, we employ the innovative Omni-dimensional Dynamic Convolution (ODConv) technology within the spectrogram input branch, enabling enhanced extraction and integration of crucial information across multiple dimensions. Additionally, we refine the Conformer model by enhancing its feature extraction capabilities not only in the temporal dimension but also across the spectral domain. Extensive experiments on the VCTK-DEMAND dataset show that HFSDA is comparable to existing state-of-the-art models, confirming the validity of our approach.
Paper Structure (13 sections, 3 equations, 3 figures, 2 tables)

This paper contains 13 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. related work
  3. Self-Supervised Learning Models
  4. Conformer
  5. method
  6. ODConv Module
  7. DDA MODULE
  8. EXPERIMENTAL SETUP
  9. Dataset and Assessment of indicators
  10. Experimental setup
  11. Performance Comparison
  12. Ablation Analysis
  13. CONCLUSIONS

Figures (3)

  • Figure 1: The architecture of our proposed method
  • Figure 2: In ODConv, four distinct types of attention multiplication are progressively applied to convolutional kernels. Specifically, (a) denotes location-wise multiplication operations along the temporal dimension, (b) represents location-wise multiplication operations along the frequency dimension, (c) indicates channel-wise multiplication operations along the output channel dimension, and (d) corresponds to kernel-wise multiplication operations along the dimension of the convolutional kernel.
  • Figure 3: The FA module initially extracts features through two weighted pooling layers, producing 1×F dimensional attention weights. These weights are then fed into a feed-forward layer and processed through a sigmoid activation function for a nonlinear transformation. After activation, the attention weights are expanded to T×F dimensions, forming a relevance score. These attention scores are element-wise multiplied with the original T×F dimensional input to optimize the weights of various parts of the original input data. This output reflects the state of the input features after being adjusted by the attention mechanism.