Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Residual Tokens Enhance Masked Autoencoders for Speech Modeling

Samir Sadok, Stéphane Lathuilière, Xavier Alameda-Pineda

TL;DR

RT-MAE tackles the limitation of attribute-only speech modeling by introducing residual tokens that capture information not explained by explicit attributes. The residuals are learned via cross-attention to a fixed, small set of queries, forming a bottleneck that enriches reconstruction and enables controlled manipulation, including robust speech denoising. A dropout-based regularization prevents overreliance on residuals, preserving controllability while enabling expressive synthesis. Empirical results on LibriSpeech and EmoV-DB show improvements in intelligibility, naturalness, and speaker/emotion preservation, and a noise-disentanglement extension demonstrates practical benefits for speech enhancement.

Abstract

Recent speech modeling relies on explicit attributes such as pitch, content, and speaker identity, but these alone cannot capture the full richness of natural speech. We introduce RT-MAE, a novel masked autoencoder framework that augments the supervised attributes-based modeling with unsupervised residual trainable tokens, designed to encode the information not explained by explicit labeled factors (e.g., timbre variations, noise, emotion etc). Experiments show that RT-MAE improves reconstruction quality, preserving content and speaker similarity while enhancing expressivity. We further demonstrate its applicability to speech enhancement, removing noise at inference while maintaining controllability and naturalness.

Residual Tokens Enhance Masked Autoencoders for Speech Modeling

TL;DR

RT-MAE tackles the limitation of attribute-only speech modeling by introducing residual tokens that capture information not explained by explicit attributes. The residuals are learned via cross-attention to a fixed, small set of queries, forming a bottleneck that enriches reconstruction and enables controlled manipulation, including robust speech denoising. A dropout-based regularization prevents overreliance on residuals, preserving controllability while enabling expressive synthesis. Empirical results on LibriSpeech and EmoV-DB show improvements in intelligibility, naturalness, and speaker/emotion preservation, and a noise-disentanglement extension demonstrates practical benefits for speech enhancement.

Abstract

Recent speech modeling relies on explicit attributes such as pitch, content, and speaker identity, but these alone cannot capture the full richness of natural speech. We introduce RT-MAE, a novel masked autoencoder framework that augments the supervised attributes-based modeling with unsupervised residual trainable tokens, designed to encode the information not explained by explicit labeled factors (e.g., timbre variations, noise, emotion etc). Experiments show that RT-MAE improves reconstruction quality, preserving content and speaker similarity while enhancing expressivity. We further demonstrate its applicability to speech enhancement, removing noise at inference while maintaining controllability and naturalness.
Paper Structure (9 sections, 3 figures, 4 tables)

This paper contains 9 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. The proposed approach: RT-MAE
  3. Background and Intuition
  4. Extracting the residual tokens via cross-attention
  5. Regularization strategy
  6. Experiments
  7. Experimental setup
  8. Results and discussion
  9. Conclusion

Figures (3)

  • Figure 1: Overview of the proposed RT-MAE architecture. The top (blue) illustrates the current paradigm, where speech and explicit attributes are jointly modeled using an MAE. The bottom (purple) illustrates the introduction of trainable queries, which through cross-attention, capture residual factors not captured by the explicit attributes.
  • Figure 1: Speech analysis and synthesis results on LibriSpeech and EmoV-DB test sets. GT MS refers to the ground-truth Mel-Spectrogram, which is converted to waveform using the HiFi-GAN vocoder (without passing through the model).
  • Figure 2: Effect of the dropout threshold $\tau$ on synthesis quality. In blue (), only the attributes are used ($\mathbf{R}$ is masked). In orange (), both attributes and residual tokens are used.