Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards noise-robust speech inversion through multi-task learning with speech enhancement

Saba Tabatabaee, Carol Espy-Wilson

TL;DR

The paper addresses the challenge of robust speech inversion (SI) in noisy environments by introducing a unified framework that jointly trains Speech Enhancement (SE) and SI using shared WavLM-Large SSL representations. It employs a Two-Stage Training (TST) strategy to stabilize learning before joint fine-tuning, and compares SI performance when SE is used as a preprocessor versus integrated in a multi-task setting (SISE-M). Across XRMB-based evaluations with babble and non-babble noise, the SISE-M approach yields notable improvements in SI (average Pearson correlation across 10 parameters) and SE metrics (PESQ, STOI, etc.), while maintaining strong performance on clean data. The findings demonstrate that SSL-based representations support simultaneous SE and SI, offering improved robustness for real-world noisy speech and revealing practical benefits for articulatory estimation and related applications.

Abstract

Recent studies demonstrate the effectiveness of Self Supervised Learning (SSL) speech representations for Speech Inversion (SI). However, applying SI in real-world scenarios remains challenging due to the pervasive presence of background noise. We propose a unified framework that integrates Speech Enhancement (SE) and SI models through shared SSL-based speech representations. In this framework, the SSL model is trained not only to support the SE module in suppressing noise but also to produce representations that are more informative for the SI task, allowing both modules to benefit from joint training. At a Signal-to-Noise Ratio of -5 db, our method for the SI task achieves relative improvements over the baseline of 80.95% under babble noise and 38.98% under non-babble noise, as measured by the average Pearson product-moment correlation across all estimated parameters.

Towards noise-robust speech inversion through multi-task learning with speech enhancement

TL;DR

The paper addresses the challenge of robust speech inversion (SI) in noisy environments by introducing a unified framework that jointly trains Speech Enhancement (SE) and SI using shared WavLM-Large SSL representations. It employs a Two-Stage Training (TST) strategy to stabilize learning before joint fine-tuning, and compares SI performance when SE is used as a preprocessor versus integrated in a multi-task setting (SISE-M). Across XRMB-based evaluations with babble and non-babble noise, the SISE-M approach yields notable improvements in SI (average Pearson correlation across 10 parameters) and SE metrics (PESQ, STOI, etc.), while maintaining strong performance on clean data. The findings demonstrate that SSL-based representations support simultaneous SE and SI, offering improved robustness for real-world noisy speech and revealing practical benefits for articulatory estimation and related applications.

Abstract

Recent studies demonstrate the effectiveness of Self Supervised Learning (SSL) speech representations for Speech Inversion (SI). However, applying SI in real-world scenarios remains challenging due to the pervasive presence of background noise. We propose a unified framework that integrates Speech Enhancement (SE) and SI models through shared SSL-based speech representations. In this framework, the SSL model is trained not only to support the SE module in suppressing noise but also to produce representations that are more informative for the SI task, allowing both modules to benefit from joint training. At a Signal-to-Noise Ratio of -5 db, our method for the SI task achieves relative improvements over the baseline of 80.95% under babble noise and 38.98% under non-babble noise, as measured by the average Pearson product-moment correlation across all estimated parameters.
Paper Structure (12 sections, 2 equations, 3 figures, 3 tables)

This paper contains 12 sections, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Data prepration
  3. XRMB dataset
  4. Data augmentation for XRMB dataset
  5. Methodology
  6. SE model
  7. SI model
  8. Results and Discussion
  9. Speech inversion
  10. Speech enhancement
  11. Performance under babble versus non-babble noise
  12. Conclusions

Figures (3)

  • Figure 1: A visual illustration of the vocal tract variables.
  • Figure 2: Architecture of the proposed SISE-M model.
  • Figure 3: Spectrograms of clean, noisy, and enhanced speech obtained from SISE-M model. The noisy speech corresponds to an XRMB test sample corrupted by babble noise at -5 dB SNR. TTCD estimates from the SI-O and SISE-M models for the noisy speech are shown alongside the ground-truth.