Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

I-DCCRN-VAE: An Improved Deep Representation Learning Framework for Complex VAE-based Single-channel Speech Enhancement

Jiatong Li, Simon Doclo

TL;DR

The paper addresses robust single-channel speech enhancement with complex VAE-based models by introducing I-DCCRN-VAE, which eliminates skip connections in pretrained VAEs, uses $eta$-VAE pretraining to balance reconstruction and latent-space regularization, and employs an NSVAE that learns both speech and noise latent representations. This design aims to mitigate posterior collapse and enhance generalization to mismatched data, while reducing reliance on adversarial training. Empirical results show comparable performance to baselines on matched DNS3 data but clear improvements on mismatched WSJ0-QUT and VB-DMD datasets, demonstrating better generalization; importantly, the method achieves these gains with classical fine-tuning, avoiding the training instability of adversarial schemes. Overall, I-DCCRN-VAE offers a simpler, more generalizable approach to complex VAE-based speech enhancement with practical implications for real-world noisy environments.

Abstract

Recently, a complex variational autoencoder (VAE)-based single-channel speech enhancement system based on the DCCRN architecture has been proposed. In this system, a noise suppression VAE (NSVAE) learns to extract clean speech representations from noisy speech using pretrained clean speech and noise VAEs with skip connections. In this paper, we improve DCCRN-VAE by incorporating three key modifications: 1) removing the skip connections in the pretrained VAEs to encourage more informative speech and noise latent representations; 2) using $β$-VAE in pretraining to better balance reconstruction and latent space regularization; and 3) a NSVAE generating both speech and noise latent representations. Experiments show that the proposed system achieves comparable performance as the DCCRN and DCCRN-VAE baselines on the matched DNS3 dataset but outperforms the baselines on mismatched datasets (WSJ0-QUT, Voicebank-DEMEND), demonstrating improved generalization ability. In addition, an ablation study shows that a similar performance can be achieved with classical fine-tuning instead of adversarial training, resulting in a simpler training pipeline.

I-DCCRN-VAE: An Improved Deep Representation Learning Framework for Complex VAE-based Single-channel Speech Enhancement

TL;DR

The paper addresses robust single-channel speech enhancement with complex VAE-based models by introducing I-DCCRN-VAE, which eliminates skip connections in pretrained VAEs, uses -VAE pretraining to balance reconstruction and latent-space regularization, and employs an NSVAE that learns both speech and noise latent representations. This design aims to mitigate posterior collapse and enhance generalization to mismatched data, while reducing reliance on adversarial training. Empirical results show comparable performance to baselines on matched DNS3 data but clear improvements on mismatched WSJ0-QUT and VB-DMD datasets, demonstrating better generalization; importantly, the method achieves these gains with classical fine-tuning, avoiding the training instability of adversarial schemes. Overall, I-DCCRN-VAE offers a simpler, more generalizable approach to complex VAE-based speech enhancement with practical implications for real-world noisy environments.

Abstract

Recently, a complex variational autoencoder (VAE)-based single-channel speech enhancement system based on the DCCRN architecture has been proposed. In this system, a noise suppression VAE (NSVAE) learns to extract clean speech representations from noisy speech using pretrained clean speech and noise VAEs with skip connections. In this paper, we improve DCCRN-VAE by incorporating three key modifications: 1) removing the skip connections in the pretrained VAEs to encourage more informative speech and noise latent representations; 2) using -VAE in pretraining to better balance reconstruction and latent space regularization; and 3) a NSVAE generating both speech and noise latent representations. Experiments show that the proposed system achieves comparable performance as the DCCRN and DCCRN-VAE baselines on the matched DNS3 dataset but outperforms the baselines on mismatched datasets (WSJ0-QUT, Voicebank-DEMEND), demonstrating improved generalization ability. In addition, an ablation study shows that a similar performance can be achieved with classical fine-tuning instead of adversarial training, resulting in a simpler training pipeline.
Paper Structure (10 sections, 8 equations, 1 figure, 2 tables)

This paper contains 10 sections, 8 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. The Proposed I-DCCRN-VAE system
  3. Signal Model
  4. System Description
  5. Experiments
  6. Training and Evaluation Datasets
  7. Network and Training
  8. Experimental Results: Hyperparameter Optimization
  9. Experimental Results: Comparison with Baselines
  10. Conclusions

Figures (1)

  • Figure 1: Overview of the improved DCCRN-VAE, consisting of two pretrained VAEs without skip connections, i.e. clean speech VAE (CVAE) and noise VAE (NVAE), and the noise suppression VAE (NSVAE).