XANE: eXplainable Acoustic Neural Embeddings

TL;DR

XANE introduces explainable neural embeddings that capture background acoustics from speech in a non-intrusive manner. Using Transformer/Conformer architectures with MelFB or WavLM features, it predicts 14 acoustic parameters and 3 classification tasks to make the embeddings interpretable. It achieves strong clustering performance with a mean F1 of $95.2\%$ across three tasks and outperforms baselines such as WavLM, TAS, and NISA+, including a $24.5\%$ relative MAE improvement on reverberation metrics and a $17\times$ faster runtime. This approach has practical implications for robust distant ASR and downstream applications like TTS, with scope for further improvement through better noise labeling.

Abstract

We present a novel method for extracting neural embeddings that model the background acoustics of a speech signal. The extracted embeddings are used to estimate specific parameters related to the background acoustic properties of the signal in a non-intrusive manner, which allows the embeddings to be explainable in terms of those parameters. We illustrate the value of these embeddings by performing clustering experiments on unseen test data and show that the proposed embeddings achieve a mean F1 score of 95.2\% for three different tasks, outperforming significantly the WavLM based signal embeddings. We also show that the proposed method can explain the embeddings by estimating 14 acoustic parameters characterizing the background acoustics, including reverberation and noise levels, overlapped speech detection, CODEC type detection and noise type detection with high accuracy and a real-time factor 17 times lower than an external baseline method.

Figures (2)

• Figure 1: XANE architecture based on Transformers and MelFB features.
• Figure 2: T-SNE tsne plots for XANE embeddings (top: overlap detection, middle: reverberant speech detection and bottom: noisy type clustering) on the ACE data.