Mask-Weighted Spatial Likelihood Coding for Speaker-Independent Joint Localization and Mask Estimation

TL;DR

This work tackles joint localization and mask estimation for speaker-independent multichannel speech by introducing mask-weighted spatial likelihood coding (mwslc). By weighting Gaussian spatial kernels with time–frequency masks, mwslc provides a continuous, well-conditioned encoding over a spatial grid, addressing gradient issues that plagued prior binary encodings. The approach enables simultaneous DoA and mask extraction and demonstrates superior performance compared with sbc, slc, and mwsbc baselines, particularly in joint estimation tasks. Experiments on MC-LibriMix show that mwslc improves both localization (DoA) and separation metrics with a unified training framework, offering a practical pathway to replace upstream ssl systems in performance-critical applications.

Abstract

Due to their robustness and flexibility, neural-driven beamformers are a popular choice for speech separation in challenging environments with a varying amount of simultaneous speakers alongside noise and reverberation. Time-frequency masks and relative directions of the speakers regarding a fixed spatial grid can be used to estimate the beamformer's parameters. To some degree, speaker-independence is achieved by ensuring a greater amount of spatial partitions than speech sources. In this work, we analyze how to encode both mask and positioning into such a grid to enable joint estimation of both quantities. We propose mask-weighted spatial likelihood coding and show that it achieves considerable performance in both tasks compared to baseline encodings optimized for either localization or mask estimation. In the same setup, we demonstrate superiority for joint estimation of both quantities. Conclusively, we propose a universal approach which can replace an upstream sound source localization system solely by adapting the training framework, making it highly relevant in performance-critical scenarios.

Figures (2)

• Figure 1: Joint localization and mask estimation framework. The estimated spatio-spectral coding $\widehat{L}_{tk\theta}$ is averaged and post-processed to obtain the time-independent doas $\widehat{\theta}^{(i)}$. Finally, the irm $\widehat{M}_{tk}^{(i)}$ are recovered by sampling $\widehat{L}_{tk\theta}$ at $\widehat{\theta}^{(i)}$.
• Figure 2: Ground truth (a, b), estimated mwsbc (c, d) and our proposed mwslc (e, f) with mccruse from top to bottom. Averaged coding $\widehat{\ell}_{t\theta}$ with clustered doas and corresponding masks $\widehat{M}^{(i)}_{tk}$ for speaker ( ) on the left and right respectively. The mask for mwsbc (d) is extracted using oracle doa.