Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MossFormer2: Combining Transformer and RNN-Free Recurrent Network for Enhanced Time-Domain Monaural Speech Separation

Shengkui Zhao, Yukun Ma, Chongjia Ni, Chong Zhang, Hao Wang, Trung Hieu Nguyen, Kun Zhou, Jiaqi Yip, Dianwen Ng, Bin Ma

TL;DR

MossFormer2 addresses the need to capture both global dependencies and fine-scale recurrent patterns in time-domain monaural speech separation by integrating a dilated FSMN-based, RNN-free recurrent module into the MossFormer framework. The hybrid architecture combines MossFormer’s joint local-global self-attention with a per-dimension recurrent learning pathway implemented via gated convolutions and dense, dilated memory blocks. Empirical results on WSJ0-2mix/3mix, Libri2Mix, and WHAM!/WHAMR! show state-of-the-art SI-SDRi improvements with substantially fewer parameters than comparable methods, highlighting the effectiveness and efficiency of the approach. This work demonstrates a practical route to robust, real-time capable speech separation by unifying global context with fine-grained temporal recurrence in a single model.

Abstract

Our previously proposed MossFormer has achieved promising performance in monaural speech separation. However, it predominantly adopts a self-attention-based MossFormer module, which tends to emphasize longer-range, coarser-scale dependencies, with a deficiency in effectively modelling finer-scale recurrent patterns. In this paper, we introduce a novel hybrid model that provides the capabilities to model both long-range, coarse-scale dependencies and fine-scale recurrent patterns by integrating a recurrent module into the MossFormer framework. Instead of applying the recurrent neural networks (RNNs) that use traditional recurrent connections, we present a recurrent module based on a feedforward sequential memory network (FSMN), which is considered "RNN-free" recurrent network due to the ability to capture recurrent patterns without using recurrent connections. Our recurrent module mainly comprises an enhanced dilated FSMN block by using gated convolutional units (GCU) and dense connections. In addition, a bottleneck layer and an output layer are also added for controlling information flow. The recurrent module relies on linear projections and convolutions for seamless, parallel processing of the entire sequence. The integrated MossFormer2 hybrid model demonstrates remarkable enhancements over MossFormer and surpasses other state-of-the-art methods in WSJ0-2/3mix, Libri2Mix, and WHAM!/WHAMR! benchmarks (https://github.com/modelscope/ClearerVoice-Studio).

MossFormer2: Combining Transformer and RNN-Free Recurrent Network for Enhanced Time-Domain Monaural Speech Separation

TL;DR

MossFormer2 addresses the need to capture both global dependencies and fine-scale recurrent patterns in time-domain monaural speech separation by integrating a dilated FSMN-based, RNN-free recurrent module into the MossFormer framework. The hybrid architecture combines MossFormer’s joint local-global self-attention with a per-dimension recurrent learning pathway implemented via gated convolutions and dense, dilated memory blocks. Empirical results on WSJ0-2mix/3mix, Libri2Mix, and WHAM!/WHAMR! show state-of-the-art SI-SDRi improvements with substantially fewer parameters than comparable methods, highlighting the effectiveness and efficiency of the approach. This work demonstrates a practical route to robust, real-time capable speech separation by unifying global context with fine-grained temporal recurrence in a single model.

Abstract

Our previously proposed MossFormer has achieved promising performance in monaural speech separation. However, it predominantly adopts a self-attention-based MossFormer module, which tends to emphasize longer-range, coarser-scale dependencies, with a deficiency in effectively modelling finer-scale recurrent patterns. In this paper, we introduce a novel hybrid model that provides the capabilities to model both long-range, coarse-scale dependencies and fine-scale recurrent patterns by integrating a recurrent module into the MossFormer framework. Instead of applying the recurrent neural networks (RNNs) that use traditional recurrent connections, we present a recurrent module based on a feedforward sequential memory network (FSMN), which is considered "RNN-free" recurrent network due to the ability to capture recurrent patterns without using recurrent connections. Our recurrent module mainly comprises an enhanced dilated FSMN block by using gated convolutional units (GCU) and dense connections. In addition, a bottleneck layer and an output layer are also added for controlling information flow. The recurrent module relies on linear projections and convolutions for seamless, parallel processing of the entire sequence. The integrated MossFormer2 hybrid model demonstrates remarkable enhancements over MossFormer and surpasses other state-of-the-art methods in WSJ0-2/3mix, Libri2Mix, and WHAM!/WHAMR! benchmarks (https://github.com/modelscope/ClearerVoice-Studio).
Paper Structure (13 sections, 2 equations, 3 figures, 4 tables)

This paper contains 13 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. The MossFormer2 Model
  3. Encoder and Decoder
  4. Hybrid MossFormer and Recurrent Modules
  5. RNN-Free Recurrent Module
  6. Dilated FSMN Block
  7. Conv-U Block
  8. Experiments
  9. Dataset
  10. Training Setup
  11. Results
  12. Ablation Studies
  13. Conclusions

Figures (3)

  • Figure 1: (A) The diagram of the base MossFormer model. (B) The diagram of the proposed MossFormer2 model. Compared to MossFormer using only MossFormer module, hybrid MossFormer and recurrent modules are adopted in MossFormer2.
  • Figure 2: Illustration of hybrid MossFormer and recurrent modules. In this setup, the MossFormer module (A) handles the complete sequence to capture global dependencies, whereas the recurrent module (B) engages in recurrent learning across each embedding dimension. ($S$: the sequence length, $N$: the embedding dimension).
  • Figure 3: (A) The flowchart of the proposed recurrent module. It comprises a bottleneck layer, a GCU layer, and an output layer. The GCU layer is built on gated convolutional units and a dillated FSMN block. (B) The design of the dilated FSMN block. (C) The design of the 2-D Conv block. (D) The design of the Conv-U block.