Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation

Min-Jae Hwang, Ilia Kulikov, Benjamin Peloquin, Hongyu Gong, Peng-Jen Chen, Ann Lee

TL;DR

This work tackles the challenge of preserving source expressivity in expressive speech-to-speech translation under real-world noise. It introduces DINO-PRETSSEL, a textless acoustic model that integrates DINO self-distillation into the PRETSSEL U2S framework to learn noise-robust expressivity embeddings. Through extensive experiments on English–Spanish and Spanish–English with synthetic and real-world noise, the method achieves superior content and prosody preservation in noisy conditions while remaining competitive in clean environments, as shown by objective metrics (ASR-BLEU, AutoPCP, SNR) and subjective MOS/S-MOS tests. The approach demonstrates practical potential for robust, natural, cross-lingual speech translation, though it entails longer pretraining times and raises considerations around biometric data usage.

Abstract

In this paper, we propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-to-speech translation (S2ST). Recently proposed expressive S2ST systems have achieved impressive expressivity preservation performances by cascading unit-to-speech (U2S) generator to the speech-to-unit translation model. However, these systems are vulnerable to the presence of noise in input speech, which is an assumption in real-world translation scenarios. To address this limitation, we propose a U2S generator that incorporates a distillation with no label (DINO) self-supervised training strategy into it's pretraining process. Because the proposed method captures noise-agnostic expressivity representation, it can generate qualified speech even in noisy environment. Objective and subjective evaluation results verified that the proposed method significantly improved the performance of the expressive S2ST system in noisy environments while maintaining competitive performance in clean environments.

Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation

TL;DR

This work tackles the challenge of preserving source expressivity in expressive speech-to-speech translation under real-world noise. It introduces DINO-PRETSSEL, a textless acoustic model that integrates DINO self-distillation into the PRETSSEL U2S framework to learn noise-robust expressivity embeddings. Through extensive experiments on English–Spanish and Spanish–English with synthetic and real-world noise, the method achieves superior content and prosody preservation in noisy conditions while remaining competitive in clean environments, as shown by objective metrics (ASR-BLEU, AutoPCP, SNR) and subjective MOS/S-MOS tests. The approach demonstrates practical potential for robust, natural, cross-lingual speech translation, though it entails longer pretraining times and raises considerations around biometric data usage.

Abstract

In this paper, we propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-to-speech translation (S2ST). Recently proposed expressive S2ST systems have achieved impressive expressivity preservation performances by cascading unit-to-speech (U2S) generator to the speech-to-unit translation model. However, these systems are vulnerable to the presence of noise in input speech, which is an assumption in real-world translation scenarios. To address this limitation, we propose a U2S generator that incorporates a distillation with no label (DINO) self-supervised training strategy into it's pretraining process. Because the proposed method captures noise-agnostic expressivity representation, it can generate qualified speech even in noisy environment. Objective and subjective evaluation results verified that the proposed method significantly improved the performance of the expressive S2ST system in noisy environments while maintaining competitive performance in clean environments.
Paper Structure (59 sections, 7 equations, 5 figures, 9 tables)

This paper contains 59 sections, 7 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Expressive S2ST
  4. Noise-robust expressive TTS
  5. Background: Expressive S2ST with PRETSSEL
  6. Architecture of PRETSSEL
  7. Pretraining
  8. Application in Expressive S2ST
  9. DINO-PRETSSEL
  10. Learning robust expressivity embedding with self-distillation
  11. Student training
  12. Teacher training
  13. Avoiding model collapse.
  14. Noise augmentation.
  15. Pretraining
  16. ...and 44 more sections

Figures (5)

  • Figure 1: Pretraining of conventional PRETSSEL (left) and proposed DINO-PRETSSEL (right). Target and predicted features include Mel-spectrograms, pitch, energy, and voicing flag.
  • Figure 2: Objective evaluation results of various expressive S2ST systems under different SNR conditions.
  • Figure 3: t-SNE plot of expressivity embeddings obtained from clean speeches.
  • Figure 4: t-SNE plot of expressivity embeddings obtained from noisy speeches.
  • Figure 5: Objective evaluation results of various PRETSSEL models with ground-truth units.