Disentangling Textual and Acoustic Features of Neural Speech Representations

TL;DR

This work tackles the entanglement of neural speech representations by introducing a two-stage Variational Information Bottleneck framework to disentangle textual (transcribable) from acoustic (non-text) information at the frame level. Stage 1 compresses NSM outputs to preserve transcription, yielding z_textual, while Stage 2 uses these textual latents to learn z_acoustic that supports a downstream task such as emotion or speaker identification; the two latent streams are then combined for decoding. Across emotion and speaker tasks, the approach achieves strong transcription and task performance with highly disentangled latent spaces, as validated by probing and qualitative analyses, and reveals layerwise dynamics of textual and acoustic information, including potential privacy benefits. The method is modality-agnostic and offers disentangled feature attribution, with implications for privacy-preserving deployments and broader applications beyond speech recognition.

Abstract

Neural speech models build deeply entangled internal representations, which capture a variety of features (e.g., fundamental frequency, loudness, syntactic category, or semantic content of a word) in a distributed encoding. This complexity makes it difficult to track the extent to which such representations rely on textual and acoustic information, or to suppress the encoding of acoustic features that may pose privacy risks (e.g., gender or speaker identity) in critical, real-world applications. In this paper, we build upon the Information Bottleneck principle to propose a disentanglement framework that separates complex speech representations into two distinct components: one encoding content (i.e., what can be transcribed as text) and the other encoding acoustic features relevant to a given downstream task. We apply and evaluate our framework to emotion recognition and speaker identification downstream tasks, quantifying the contribution of textual and acoustic features at each model layer. Additionally, we explore the application of our disentanglement framework as an attribution method to identify the most salient speech frame representations from both the textual and acoustic perspectives.

Figures (13)

• Figure 1: An overview of our disentanglement framework: In stage 1, hidden states derived from a frozen speech model are compressed using VIB to decode transcription. In stage 2, the hidden states are compressed to decode emotion, conditioned on the frozen textual latent representations learned in stage 1. This procedure results in disentangled latent representations for textual and acoustic features relevant to the target task within each speech frame representation.
• Figure 2: Probing performances of latent representations ($d\!=\!128$) learned at stages 1 and 2, along with hidden states derived from Large models for transcription and a set of audio features. For the WER metric, the lower score is better, while for other metrics, the higher is better. The dashed line in each plot represents the random baseline.
• Figure 3: t-SNE plots of the textual and acoustic latent representations for Wav2Vec2-Large, marked and colored according to their transcription and emotion labels, respectively.
• Figure 4: Layerwise performance of VIB ($d\!=\!128$) for transcription at stage 1 ( left) and emotion classification at stage 2 ( middle), compared to layerwise performance using original hidden state activations. Right: layerwise textual and acoustic contribution to emotion classification, compared to layerwise performance using hidden states of Base models. Lower WER and higher accuracy are better. The black horizontal dashed line represents the random baseline.
• Figure 5: Dot product of attribution scores and different acoustic and textual cues.