Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Time-Frequency-Based Attention Cache Memory Model for Real-Time Speech Separation

Guo Chen, Kai Li, Runxuan Yang, Xiaolin Hu

TL;DR

Real-time speech separation must operate with only past context, which degrades performance compared with offline approaches. The authors propose TFACM, a causal, attention-based TF-domain framework that uses an LSTM to model frequency-relativ positions, local/global time segmentation, and a Cache Memory to store past hidden states, augmented by a CAR module to refine attention with history. Empirical results on WHAM!, WHAMR!, and LibriMix at 8 kHz show TFACM Large achieves performance on par with TF-GridNet-Causal but with only 8.8% of the parameters and 20.4% of MACs, while TFACM Small further reduces latency. The work demonstrates the value of explicitly storing and reusing historical information in causal TF models, enabling practical real-time deployment in challenging acoustics.

Abstract

Existing causal speech separation models often underperform compared to non-causal models due to difficulties in retaining historical information. To address this, we propose the Time-Frequency Attention Cache Memory (TFACM) model, which effectively captures spatio-temporal relationships through an attention mechanism and cache memory (CM) for historical information storage. In TFACM, an LSTM layer captures frequency-relative positions, while causal modeling is applied to the time dimension using local and global representations. The CM module stores past information, and the causal attention refinement (CAR) module further enhances time-based feature representations for finer granularity. Experimental results showed that TFACM achieveed comparable performance to the SOTA TF-GridNet-Causal model, with significantly lower complexity and fewer trainable parameters. For more details, visit the project page: https://cslikai.cn/TFACM/.

Time-Frequency-Based Attention Cache Memory Model for Real-Time Speech Separation

TL;DR

Real-time speech separation must operate with only past context, which degrades performance compared with offline approaches. The authors propose TFACM, a causal, attention-based TF-domain framework that uses an LSTM to model frequency-relativ positions, local/global time segmentation, and a Cache Memory to store past hidden states, augmented by a CAR module to refine attention with history. Empirical results on WHAM!, WHAMR!, and LibriMix at 8 kHz show TFACM Large achieves performance on par with TF-GridNet-Causal but with only 8.8% of the parameters and 20.4% of MACs, while TFACM Small further reduces latency. The work demonstrates the value of explicitly storing and reusing historical information in causal TF models, enabling practical real-time deployment in challenging acoustics.

Abstract

Existing causal speech separation models often underperform compared to non-causal models due to difficulties in retaining historical information. To address this, we propose the Time-Frequency Attention Cache Memory (TFACM) model, which effectively captures spatio-temporal relationships through an attention mechanism and cache memory (CM) for historical information storage. In TFACM, an LSTM layer captures frequency-relative positions, while causal modeling is applied to the time dimension using local and global representations. The CM module stores past information, and the causal attention refinement (CAR) module further enhances time-based feature representations for finer granularity. Experimental results showed that TFACM achieveed comparable performance to the SOTA TF-GridNet-Causal model, with significantly lower complexity and fewer trainable parameters. For more details, visit the project page: https://cslikai.cn/TFACM/.
Paper Structure (14 sections, 2 equations, 3 figures, 2 tables)

This paper contains 14 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. TF-based Attention Cache Memory Model
  3. Model Overview
  4. A Naive Extension to TF-GridNet
  5. Separator of TFACM
  6. Experiment configurations
  7. Dataset
  8. Model configurations
  9. Training and evaluation
  10. Results
  11. Comparison with SOTA methods
  12. Comparison on different audio input length
  13. Ablation Studies on the CM and CAR Modules
  14. Conclusion

Figures (3)

  • Figure 1: The pipeline of TFACM.
  • Figure 2: The pipeline of the TFACM separator. PW/DW-Conv represents the point-wise/depth-wise convolutional layer.
  • Figure 3: Comparison with longer audio input at the training stage and the testing stage.