Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpatialEmb: Extract and Encode Spatial Information for 1-Stage Multi-channel Multi-speaker ASR on Arbitrary Microphone Arrays

Yiwen Shao, Yong Xu, Sanjeev Khudanpur, Dong Yu

TL;DR

SpatialEmb introduces a 1-stage multi-channel ASR framework that directly encodes spatial information into the ASR model to handle arbitrary microphone arrays. It combines spectral features with a spatial embedding module (Conv2d, ConvNext, or GRU-Conv2d) and introduces a parameter-free Divide-Average-Concatenate (DAC) mechanism to support topology-agnostic processing. Through extensive experiments on AliMeeting, SpatialEmb with GRU-Conv2d achieves state-of-the-art CERs of 17.04% on Eval and 20.32% on Test using only 105 hours of training data, outperforming both fixed-topology and prior 1-stage systems. The approach offers a practical, efficient path toward universal multi-channel ASR across devices by reducing preprocessing, leveraging spatial cues, and enabling flexible microphone configurations.

Abstract

Spatial information is a critical clue for multi-channel multi-speaker target speech recognition. Most state-of-the-art multi-channel Automatic Speech Recognition (ASR) systems extract spatial features only during the speech separation stage, followed by standard single-channel ASR on the separated speech. This approach results in an inefficient, lengthy pipeline and sub-optimal ASR performance due to the accumulated errors from preprocessing modules. Furthermore, most spatial feature extraction methods depend on the knowledge of speaker positions and microphone topology, making the systems reliant on specific settings and challenging to adapt to new equipment. In this work, we propose a solution to these issues with a lightweight embedding module named SpatialEmb, which extracts and encodes spatial information directly for the ASR model, supporting both fixed and arbitrary microphone topology. We conduct comprehensive experiments on AliMeeting, a real meeting corpus, to determine the optimal model design for SpatialEmb in terms of both performance and efficiency. Our best model trained with 105 hours Train-Ali-far achieves 17.04% and 20.32% character error rates (CER) on the Eval and Test sets, establishing a new state-of-the-art result with the same training data.

SpatialEmb: Extract and Encode Spatial Information for 1-Stage Multi-channel Multi-speaker ASR on Arbitrary Microphone Arrays

TL;DR

SpatialEmb introduces a 1-stage multi-channel ASR framework that directly encodes spatial information into the ASR model to handle arbitrary microphone arrays. It combines spectral features with a spatial embedding module (Conv2d, ConvNext, or GRU-Conv2d) and introduces a parameter-free Divide-Average-Concatenate (DAC) mechanism to support topology-agnostic processing. Through extensive experiments on AliMeeting, SpatialEmb with GRU-Conv2d achieves state-of-the-art CERs of 17.04% on Eval and 20.32% on Test using only 105 hours of training data, outperforming both fixed-topology and prior 1-stage systems. The approach offers a practical, efficient path toward universal multi-channel ASR across devices by reducing preprocessing, leveraging spatial cues, and enabling flexible microphone configurations.

Abstract

Spatial information is a critical clue for multi-channel multi-speaker target speech recognition. Most state-of-the-art multi-channel Automatic Speech Recognition (ASR) systems extract spatial features only during the speech separation stage, followed by standard single-channel ASR on the separated speech. This approach results in an inefficient, lengthy pipeline and sub-optimal ASR performance due to the accumulated errors from preprocessing modules. Furthermore, most spatial feature extraction methods depend on the knowledge of speaker positions and microphone topology, making the systems reliant on specific settings and challenging to adapt to new equipment. In this work, we propose a solution to these issues with a lightweight embedding module named SpatialEmb, which extracts and encodes spatial information directly for the ASR model, supporting both fixed and arbitrary microphone topology. We conduct comprehensive experiments on AliMeeting, a real meeting corpus, to determine the optimal model design for SpatialEmb in terms of both performance and efficiency. Our best model trained with 105 hours Train-Ali-far achieves 17.04% and 20.32% character error rates (CER) on the Eval and Test sets, establishing a new state-of-the-art result with the same training data.
Paper Structure (28 sections, 7 equations, 4 figures, 4 tables)

This paper contains 28 sections, 7 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Feature Extraction: Spectral and Spatial Features
  3. Spectral feature
  4. Spatial feature
  5. Feature fusion
  6. Fixed microphone topology setting
  7. Arbitrary microphone topology setting
  8. Systems for comparison
  9. Multi-stage system
  10. Pipeline system
  11. End-to-end system
  12. 1-stage system
  13. Proposed: SpatialEmb
  14. System Overview
  15. SpatialEmb Module Structure
  16. ...and 13 more sections

Figures (4)

  • Figure 1: An illustration of (a) Multi-stage ASR system and (b) 1-stage ASR system
  • Figure 2: System overview of the proposed 1-stage ASR system using SpatialEmb.
  • Figure 3: Three types of SpatialEmb module structures.
  • Figure 4: Methods under the "Spatial Feature Expansion" category for arbitrary microphone topology, on top of the Conv2d SpatialEmb structure.