Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-channel multi-speaker transformer for speech recognition

Guo Yifan, Tian Yao, Suo Hongbin, Wan Yulong

TL;DR

The paper tackles far-field multi-speaker ASR by moving beyond separation-frontend pipelines and the existing multi-channel transformer (MCT). It introduces M2Former, an end-to-end encoder that directly learns speaker-specific acoustic embeddings from mixtures using CNN-based decoupling (CNNDD), a cross-channel attention mechanism (M2A) that leverages inter-channel similarity, and a clustering/ Filtering (CF) layer to isolate speech sources via spectral clustering and the IFSD metric. The approach achieves substantial relative word error rate reductions over neural beamformers, MCT, and other baselines on SMS-WSJ SMS-2, demonstrating that end-to-end, speaker-aware representations can be effectively learned from mixed multi-channel inputs. The model also shows robustness to unknown numbers of speakers, suggesting strong applicability to real-world teleconferencing and in-vehicle systems, with practical advantages in avoiding mismatches between frontend and backend components.

Abstract

With the development of teleconferencing and in-vehicle voice assistants, far-field multi-speaker speech recognition has become a hot research topic. Recently, a multi-channel transformer (MCT) has been proposed, which demonstrates the ability of the transformer to model far-field acoustic environments. However, MCT cannot encode high-dimensional acoustic features for each speaker from mixed input audio because of the interference between speakers. Based on these, we propose the multi-channel multi-speaker transformer (M2Former) for far-field multi-speaker ASR in this paper. Experiments on the SMS-WSJ benchmark show that the M2Former outperforms the neural beamformer, MCT, dual-path RNN with transform-average-concatenate and multi-channel deep clustering based end-to-end systems by 9.2%, 14.3%, 24.9%, and 52.2% respectively, in terms of relative word error rate reduction.

Multi-channel multi-speaker transformer for speech recognition

TL;DR

The paper tackles far-field multi-speaker ASR by moving beyond separation-frontend pipelines and the existing multi-channel transformer (MCT). It introduces M2Former, an end-to-end encoder that directly learns speaker-specific acoustic embeddings from mixtures using CNN-based decoupling (CNNDD), a cross-channel attention mechanism (M2A) that leverages inter-channel similarity, and a clustering/ Filtering (CF) layer to isolate speech sources via spectral clustering and the IFSD metric. The approach achieves substantial relative word error rate reductions over neural beamformers, MCT, and other baselines on SMS-WSJ SMS-2, demonstrating that end-to-end, speaker-aware representations can be effectively learned from mixed multi-channel inputs. The model also shows robustness to unknown numbers of speakers, suggesting strong applicability to real-world teleconferencing and in-vehicle systems, with practical advantages in avoiding mismatches between frontend and backend components.

Abstract

With the development of teleconferencing and in-vehicle voice assistants, far-field multi-speaker speech recognition has become a hot research topic. Recently, a multi-channel transformer (MCT) has been proposed, which demonstrates the ability of the transformer to model far-field acoustic environments. However, MCT cannot encode high-dimensional acoustic features for each speaker from mixed input audio because of the interference between speakers. Based on these, we propose the multi-channel multi-speaker transformer (M2Former) for far-field multi-speaker ASR in this paper. Experiments on the SMS-WSJ benchmark show that the M2Former outperforms the neural beamformer, MCT, dual-path RNN with transform-average-concatenate and multi-channel deep clustering based end-to-end systems by 9.2%, 14.3%, 24.9%, and 52.2% respectively, in terms of relative word error rate reduction.
Paper Structure (19 sections, 5 equations, 3 figures, 4 tables)

This paper contains 19 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Channel Embedding
  4. CNN Decoupling and Downsampling Layer
  5. Multi-Channel Multi-Speaker Attention Block
  6. Clustering and Filtering Layer
  7. Experiments
  8. Datasets
  9. Configurations of Baselines and the Proposed Model
  10. Separation Frontends of Baselines
  11. ASR Backend of Baselines
  12. The Proposed Model
  13. Loss Function of M2Former and the Baselines
  14. Experimental Results
  15. Comparison between M2Former and the Baselines
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Multi-channel multi-speaker transformer. $C$ is the number of input channels. $N_\text{M}$ and $N_\text{D}$ represent the numbers of M2A blocks and decoder blocks respectively. Assume that there are two speakers in the input audio.
  • Figure 2: Multi-channel attention blocks: (a)Intra-channel attention. (b) MCT's cross-channel attention. (c) M2A's cross-channel attention. $\bigotimes$, $\bigoplus$ and dashed lines denote matrix multiplication, averaging on time dimension and matrix transpose respectively.
  • Figure 3: Clustering and filtering layer. Gradients are propagated only in the solid line. Assume that there are 4 channels and 2 speakers, and the gray label get the lowest IFSD value.