Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Loose coupling of spectral and spatial models for multi-channel diarization and enhancement of meetings in dynamic environments

Adrian Meise, Tobias Cord-Landwehr, Christoph Boeddeker, Marc Delcroix, Tomohiro Nakatani, Reinhold Haeb-Umbach

TL;DR

The paper introduces a loosely coupled spectral-spatial mixture framework for joint diarization and multi-channel speech enhancement, addressing moving speakers by linking separate spectral and spatial latent variables with coupling weights. Compared to a tightly integrated model, it demonstrates improved robustness to speaker position changes on LibriCSS, particularly with longer segments and when oracle diarization is available, while remaining challenged by heavy overlaps. The approach avoids training requirements and supports arbitrary microphone arrays, advancing robust meeting transcription in dynamic environments. Practical impact lies in more reliable diarization and separation in real-world meetings where speakers move and reflections complicate localization.

Abstract

Sound capture by microphone arrays opens the possibility to exploit spatial, in addition to spectral, information for diarization and signal enhancement, two important tasks in meeting transcription. However, there is no one-to-one mapping of positions in space to speakers if speakers move. Here, we address this by proposing a novel joint spatial and spectral mixture model, whose two submodels are loosely coupled by modeling the relationship between speaker and position index probabilistically. Thus, spatial and spectral information can be jointly exploited, while at the same time allowing for speakers speaking from different positions. Experiments on the LibriCSS data set with simulated speaker position changes show great improvements over tightly coupled subsystems.

Loose coupling of spectral and spatial models for multi-channel diarization and enhancement of meetings in dynamic environments

TL;DR

The paper introduces a loosely coupled spectral-spatial mixture framework for joint diarization and multi-channel speech enhancement, addressing moving speakers by linking separate spectral and spatial latent variables with coupling weights. Compared to a tightly integrated model, it demonstrates improved robustness to speaker position changes on LibriCSS, particularly with longer segments and when oracle diarization is available, while remaining challenged by heavy overlaps. The approach avoids training requirements and supports arbitrary microphone arrays, advancing robust meeting transcription in dynamic environments. Practical impact lies in more reliable diarization and separation in real-world meetings where speakers move and reflections complicate localization.

Abstract

Sound capture by microphone arrays opens the possibility to exploit spatial, in addition to spectral, information for diarization and signal enhancement, two important tasks in meeting transcription. However, there is no one-to-one mapping of positions in space to speakers if speakers move. Here, we address this by proposing a novel joint spatial and spectral mixture model, whose two submodels are loosely coupled by modeling the relationship between speaker and position index probabilistically. Thus, spatial and spectral information can be jointly exploited, while at the same time allowing for speakers speaking from different positions. Experiments on the LibriCSS data set with simulated speaker position changes show great improvements over tightly coupled subsystems.
Paper Structure (9 sections, 10 equations, 2 figures, 2 tables)

This paper contains 9 sections, 10 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Mixture Model-based integration of separation & diarization
  3. Loose coupling of mixture models
  4. Mask estimation for loosely coupled models
  5. Evaluation
  6. Evaluation in the presence of speaker position changes
  7. Performance in a stationary environment
  8. Conclusion
  9. Acknowledgments

Figures (2)

  • Figure 1: Graphical model of the tight integration from cord2025tightintegration. The spectral model (left) and the spatial model (right) are coupled through the common latent variable $z_{ktf}$.
  • Figure 2: Graphical model of the loose coupling. The latent variable $z_{kt}$ of the spectral model serves as prior for the latent variables $z_{ltf}$ of the spatial models, which are fitted per frequency $f\in\{1,\dots, F\}$.