Masked Autoencoders as Universal Speech Enhancer
Rajalaxmi Rajagopalan, Ritwik Giri, Zhiqiang Tang, Kyu Han
TL;DR
This work introduces a universal, self-supervised speech enhancement framework built on a Masked Autoencoder (MAE) that is agnostic to distortion types and capable of handling multiple interferences. A comprehensive augmentation stack, including distance-based multi-speaker mixtures and spectrogram masking, drives a regression-based pretraining objective, enabling distortion removal and masked-region reconstruction without clean references. Finetuning on small paired datasets uses pre-trained encoder embeddings to produce a TF mask that cleans speech, with experiments showing strong performance on in-domain and out-of-domain data, especially when combining multi-speaker augmentation and log1p feature compression. Overall, the approach provides a robust foundation model for universal speech enhancement, outperforming several SSL baselines and competing state-of-the-art methods while offering broad applicability to related downstream tasks such as dereverberation and bandwidth extension.
Abstract
Supervised speech enhancement methods have been very successful. However, in practical scenarios, there is a lack of clean speech, and self-supervised learning-based (SSL) speech enhancement methods that offer comparable enhancement performance and can be applied to other speech-related downstream applications are desired. In this work, we develop a masked autoencoder based universal speech enhancer that is agnostic to the type of distortion affecting speech, can handle multiple distortions simultaneously, and is trained in a self-supervised manner. An augmentation stack adds further distortions to the noisy input data. The masked autoencoder model learns to remove the added distortions along with reconstructing the masked regions of the spectrogram during pre-training. The pre-trained embeddings are then used by fine-tuning models trained on a small amount of paired data for specific downstream tasks. We evaluate the pre-trained features for denoising and dereverberation downstream tasks. We explore different augmentations (like single or multi-speaker) in the pre-training augmentation stack and the effect of different noisy input feature representations (like $log1p$ compression) on pre-trained embeddings and downstream fine-tuning enhancement performance. We show that the proposed method not only outperforms the baseline but also achieves state-of-the-art performance for both in-domain and out-of-domain evaluation datasets.