Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Audio-Visual Speech Enhancement: Correcting Misassignments in Complex Environments with Advanced Post-Processing

Wenze Ren, Kuo-Hsuan Hung, Rong Chao, YouJin Li, Hsin-Min Wang, Yu Tsao

TL;DR

Experimental results on the AVSE-challenge dataset show that integrating PPC into the AVSE model can significantly improve AVSE performance, and combining PPC with the AVSE model trained with permutation invariant training (PIT) yields the best performance.

Abstract

This paper addresses the prevalent issue of incorrect speech output in audio-visual speech enhancement (AVSE) systems, which is often caused by poor video quality and mismatched training and test data. We introduce a post-processing classifier (PPC) to rectify these erroneous outputs, ensuring that the enhanced speech corresponds accurately to the intended speaker. We also adopt a mixup strategy in PPC training to improve its robustness. Experimental results on the AVSE-challenge dataset show that integrating PPC into the AVSE model can significantly improve AVSE performance, and combining PPC with the AVSE model trained with permutation invariant training (PIT) yields the best performance. The proposed method substantially outperforms the baseline model by a large margin. This work highlights the potential for broader applications across various modalities and architectures, providing a promising direction for future research in this field.

Robust Audio-Visual Speech Enhancement: Correcting Misassignments in Complex Environments with Advanced Post-Processing

TL;DR

Experimental results on the AVSE-challenge dataset show that integrating PPC into the AVSE model can significantly improve AVSE performance, and combining PPC with the AVSE model trained with permutation invariant training (PIT) yields the best performance.

Abstract

This paper addresses the prevalent issue of incorrect speech output in audio-visual speech enhancement (AVSE) systems, which is often caused by poor video quality and mismatched training and test data. We introduce a post-processing classifier (PPC) to rectify these erroneous outputs, ensuring that the enhanced speech corresponds accurately to the intended speaker. We also adopt a mixup strategy in PPC training to improve its robustness. Experimental results on the AVSE-challenge dataset show that integrating PPC into the AVSE model can significantly improve AVSE performance, and combining PPC with the AVSE model trained with permutation invariant training (PIT) yields the best performance. The proposed method substantially outperforms the baseline model by a large margin. This work highlights the potential for broader applications across various modalities and architectures, providing a promising direction for future research in this field.
Paper Structure (21 sections, 5 equations, 4 figures, 2 tables)

This paper contains 21 sections, 5 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Propose Method
  3. The AVSE Model
  4. Audio Feature Extraction
  5. Visual Feature Extraction
  6. Fusion
  7. The Separation Network
  8. Loss Functions
  9. The Post-processing Classifier (PPC)
  10. Model Architecture
  11. Mixup
  12. Permutation Invariant Training
  13. Experimental Setup
  14. Dataset
  15. Training and validation dataset
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Overview of the proposed audio-visual speech enhancement system.
  • Figure 2: Illustration of three different PPC models. $\otimes$ denotes the concatenation operation.
  • Figure 3: Detailed components of the audio encoder (a) and the visual encoder (b) in the PPC models, and the SE-Res2Block used in both encoders (c). Modules with a gray background are pre-trained and frozen during model training.
  • Figure 4: Scatter plot of SI-SDR scores. (a) SI-SDR scores of predicted target audio (SI-SDR$_{pt}$) and interference audio (SI-SDR$_{pi}$) produced by different AVSE models. (b) Difference in SI-SDR (SI-SDR$_{Diff.}$) before and after applying the best PPC model (PPC-3 with mixup in Table \ref{['table:acc']}).