Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Speech Representation Aggregation in Speech Enhancement: A Phonetic Mutual Information Perspective

Seungu Han, Sungho Lee, Kyogu Lee

TL;DR

This paper tackles the mismatch between self-supervised speech representations and noise-robust linguistic preservation in speech enhancement. It uses an information-theoretic analysis to quantify how much phonetic information remains in SSL features under noise via a MI bound $I(Z;Y) \ge H(Y) - \mathbb{E}_{p}[-\log q_\phi(y|z)]$. The authors propose a Linguistic Aggregation Layer pre-trained to maximize MI with phoneme labels, then frozen during SE training, and they explore a time-varying Dynamic Weighted-Sum. Experiments show that this decoupled, linguistically-aligned conditioning reduces Word Error Rate while maintaining acoustic quality, demonstrating the value of aligning conditioning modules with semantic content.

Abstract

Recent speech enhancement (SE) models increasingly leverage self-supervised learning (SSL) representations for their rich semantic information. Typically, intermediate features are aggregated into a single representation via a lightweight adaptation module. However, most SSL models are not trained for noise robustness, which can lead to corrupted semantic representations. Moreover, the adaptation module is trained jointly with the SE model, potentially prioritizing acoustic details over semantic information, contradicting the original purpose. To address this issue, we first analyze the behavior of SSL models on noisy speech from an information-theoretic perspective. Specifically, we measure the mutual information (MI) between the corrupted SSL representations and the corresponding phoneme labels, focusing on preservation of linguistic contents. Building upon this analysis, we introduce the linguistic aggregation layer, which is pre-trained to maximize MI with phoneme labels (with optional dynamic aggregation) and then frozen during SE training. Experiments show that this decoupled approach improves Word Error Rate (WER) over jointly optimized baselines, demonstrating the benefit of explicitly aligning the adaptation module with linguistic contents.

Rethinking Speech Representation Aggregation in Speech Enhancement: A Phonetic Mutual Information Perspective

TL;DR

This paper tackles the mismatch between self-supervised speech representations and noise-robust linguistic preservation in speech enhancement. It uses an information-theoretic analysis to quantify how much phonetic information remains in SSL features under noise via a MI bound . The authors propose a Linguistic Aggregation Layer pre-trained to maximize MI with phoneme labels, then frozen during SE training, and they explore a time-varying Dynamic Weighted-Sum. Experiments show that this decoupled, linguistically-aligned conditioning reduces Word Error Rate while maintaining acoustic quality, demonstrating the value of aligning conditioning modules with semantic content.

Abstract

Recent speech enhancement (SE) models increasingly leverage self-supervised learning (SSL) representations for their rich semantic information. Typically, intermediate features are aggregated into a single representation via a lightweight adaptation module. However, most SSL models are not trained for noise robustness, which can lead to corrupted semantic representations. Moreover, the adaptation module is trained jointly with the SE model, potentially prioritizing acoustic details over semantic information, contradicting the original purpose. To address this issue, we first analyze the behavior of SSL models on noisy speech from an information-theoretic perspective. Specifically, we measure the mutual information (MI) between the corrupted SSL representations and the corresponding phoneme labels, focusing on preservation of linguistic contents. Building upon this analysis, we introduce the linguistic aggregation layer, which is pre-trained to maximize MI with phoneme labels (with optional dynamic aggregation) and then frozen during SE training. Experiments show that this decoupled approach improves Word Error Rate (WER) over jointly optimized baselines, demonstrating the benefit of explicitly aligning the adaptation module with linguistic contents.
Paper Structure (12 sections, 3 equations, 3 figures, 2 tables)

This paper contains 12 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Measuring Self-supervised Models' Robustness with Mutual Information
  3. Bounding Mutual Information with Labeled Data
  4. Empirical Analysis
  5. Decoupled Linguistic Optimization for Speech Enhancement
  6. Experimental Setup
  7. Results
  8. MI-optimized Weights Analysis
  9. Speech Enhancement Results
  10. Effects of Dynamic Aggregation Strategy
  11. Discussion
  12. Acknowledgements

Figures (3)

  • Figure 1: Analysis of the MI lower bounds between the SSL features and phonemes across different layers and SNR conditions. In (a), "Average MI Lower Bound" denotes values averaged across all evaluated SNR levels.
  • Figure 2: Layer aggregation weights used for each model.
  • Figure 3: The top two subfigures show the Mel spectrogram of noisy speech and its corresponding time-varying SNR. The bottom two visualize dynamic layer weights for WavLM: (a) acoustic-tuned baseline and (b) linguistically-tuned module, respectively.