Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speech Enhancement with Overlapped-Frame Information Fusion and Causal Self-Attention

Yuewei Zhang, Huanbin Zou, Jie Zhu

TL;DR

This work tackles the inherent algorithmic delay in causal TF-domain speech enhancement caused by overlap-add in the inverse transform. It introduces OFIF, which injects pseudo future-frame information to enrich current-frame representations, and a causal TFCA module that performs time-, frequency-, and channel-wise self-attention within a CRN-based backbone. The combined OFIF-Net demonstrates state-of-the-art performance on VoiceBank+DEMAND and DNS-Challenge in causal settings, while maintaining a relatively small model size. This approach enables more effective real-time SE by leveraging delay-informed information fusion and multi-dimensional attention without violating causality.

Abstract

For time-frequency (TF) domain speech enhancement (SE) methods, the overlap-and-add operation in the inverse TF transformation inevitably leads to an algorithmic delay equal to the window size. However, typical causal SE systems fail to utilize the future speech information within this inherent delay, thereby limiting SE performance. In this paper, we propose an overlapped-frame information fusion scheme. At each frame index, we construct several pseudo overlapped-frames, fuse them with the original speech frame, and then send the fused results to the SE model. Additionally, we introduce a causal time-frequency-channel attention (TFCA) block to boost the representation capability of the neural network. This block parallelly processes the intermediate feature maps through self-attention-based operations in the time, frequency, and channel dimensions. Experiments demonstrate the superiority of these improvements, and the proposed SE system outperforms the current advanced methods.

Speech Enhancement with Overlapped-Frame Information Fusion and Causal Self-Attention

TL;DR

This work tackles the inherent algorithmic delay in causal TF-domain speech enhancement caused by overlap-add in the inverse transform. It introduces OFIF, which injects pseudo future-frame information to enrich current-frame representations, and a causal TFCA module that performs time-, frequency-, and channel-wise self-attention within a CRN-based backbone. The combined OFIF-Net demonstrates state-of-the-art performance on VoiceBank+DEMAND and DNS-Challenge in causal settings, while maintaining a relatively small model size. This approach enables more effective real-time SE by leveraging delay-informed information fusion and multi-dimensional attention without violating causality.

Abstract

For time-frequency (TF) domain speech enhancement (SE) methods, the overlap-and-add operation in the inverse TF transformation inevitably leads to an algorithmic delay equal to the window size. However, typical causal SE systems fail to utilize the future speech information within this inherent delay, thereby limiting SE performance. In this paper, we propose an overlapped-frame information fusion scheme. At each frame index, we construct several pseudo overlapped-frames, fuse them with the original speech frame, and then send the fused results to the SE model. Additionally, we introduce a causal time-frequency-channel attention (TFCA) block to boost the representation capability of the neural network. This block parallelly processes the intermediate feature maps through self-attention-based operations in the time, frequency, and channel dimensions. Experiments demonstrate the superiority of these improvements, and the proposed SE system outperforms the current advanced methods.
Paper Structure (13 sections, 9 equations, 3 figures, 3 tables)

This paper contains 13 sections, 9 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overlapped-frame information fusion (OFIF)
  4. Network structure
  5. Convolutional recurrent network (CRN)-based structure
  6. Causal Time-frequency-Channel attention (TFCA) block
  7. Loss function
  8. Experiments
  9. Datasets
  10. Experimental setup
  11. Ablation study
  12. Comparison with existing advanced methods
  13. Conclusions

Figures (3)

  • Figure 1: Time-frequency transformation & Inverse time-frequency transformation.
  • Figure 2: Overall architecture of the proposed SE system combining the overlapped-frame information fusion scheme with self-attention-based DCTCRN (OFIF-Net).
  • Figure 3: Details of causal time-frequency-channel attention (TFCA) block. The "T-Branch", "F-Branch", and "C-Branch" respectively denote the time-wise, frequency-wise, and channel-wise self-attention branches.