Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CVSS Corpus and Massively Multilingual Speech-to-Speech Translation

Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, Heiga Zen

TL;DR

This work introduces CVSS, a massively multilingual-to-English speech-to-speech translation corpus derived from Common Voice and CoVoST 2, with two translation-speech variants: CVSS-C (canonical voice) and CVSS-T (voice cloned from the source). By presenting both direct S2ST and cascade S2ST baselines, and showing that pre-training on CoVoST 2 further boosts performance, the paper demonstrates the corpus’s effectiveness for advancing end-to-end S2ST research and voice-preservation capabilities. The findings indicate that direct S2ST can approach Cascade baselines with proper initialization, and CVSS-T enables voice-preserving translation, making the dataset a valuable resource for multilingual S2ST and voice-transfer research. The release, under CC BY 4.0, aims to accelerate progress in robust, user-facing S2ST systems and cross-lingual voice transfer.

Abstract

We introduce CVSS, a massively multilingual-to-English speech-to-speech translation (S2ST) corpus, covering sentence-level parallel S2ST pairs from 21 languages into English. CVSS is derived from the Common Voice speech corpus and the CoVoST 2 speech-to-text translation (ST) corpus, by synthesizing the translation text from CoVoST 2 into speech using state-of-the-art TTS systems. Two versions of translation speeches are provided: 1) CVSS-C: All the translation speeches are in a single high-quality canonical voice; 2) CVSS-T: The translation speeches are in voices transferred from the corresponding source speeches. In addition, CVSS provides normalized translation text which matches the pronunciation in the translation speech. On each version of CVSS, we built baseline multilingual direct S2ST models and cascade S2ST models, verifying the effectiveness of the corpus. To build strong cascade S2ST baselines, we trained an ST model on CoVoST 2, which outperforms the previous state-of-the-art trained on the corpus without extra data by 5.8 BLEU. Nevertheless, the performance of the direct S2ST models approaches the strong cascade baselines when trained from scratch, and with only 0.1 or 0.7 BLEU difference on ASR transcribed translation when initialized from matching ST models.

CVSS Corpus and Massively Multilingual Speech-to-Speech Translation

TL;DR

This work introduces CVSS, a massively multilingual-to-English speech-to-speech translation corpus derived from Common Voice and CoVoST 2, with two translation-speech variants: CVSS-C (canonical voice) and CVSS-T (voice cloned from the source). By presenting both direct S2ST and cascade S2ST baselines, and showing that pre-training on CoVoST 2 further boosts performance, the paper demonstrates the corpus’s effectiveness for advancing end-to-end S2ST research and voice-preservation capabilities. The findings indicate that direct S2ST can approach Cascade baselines with proper initialization, and CVSS-T enables voice-preserving translation, making the dataset a valuable resource for multilingual S2ST and voice-transfer research. The release, under CC BY 4.0, aims to accelerate progress in robust, user-facing S2ST systems and cross-lingual voice transfer.

Abstract

We introduce CVSS, a massively multilingual-to-English speech-to-speech translation (S2ST) corpus, covering sentence-level parallel S2ST pairs from 21 languages into English. CVSS is derived from the Common Voice speech corpus and the CoVoST 2 speech-to-text translation (ST) corpus, by synthesizing the translation text from CoVoST 2 into speech using state-of-the-art TTS systems. Two versions of translation speeches are provided: 1) CVSS-C: All the translation speeches are in a single high-quality canonical voice; 2) CVSS-T: The translation speeches are in voices transferred from the corresponding source speeches. In addition, CVSS provides normalized translation text which matches the pronunciation in the translation speech. On each version of CVSS, we built baseline multilingual direct S2ST models and cascade S2ST models, verifying the effectiveness of the corpus. To build strong cascade S2ST baselines, we trained an ST model on CoVoST 2, which outperforms the previous state-of-the-art trained on the corpus without extra data by 5.8 BLEU. Nevertheless, the performance of the direct S2ST models approaches the strong cascade baselines when trained from scratch, and with only 0.1 or 0.7 BLEU difference on ASR transcribed translation when initialized from matching ST models.
Paper Structure (38 sections, 3 figures, 10 tables)

This paper contains 38 sections, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Synthetic vs Human-recorded
  4. Text normalization
  5. Source corpora
  6. Common Voice
  7. CoVoST 2
  8. TTS models
  9. PnG NAT
  10. Performance
  11. PnG NAT with voice cloning
  12. Speaker encoder
  13. Performance
  14. Data generation
  15. Data filtering
  16. ...and 23 more sections

Figures (3)

  • Figure 1: PnG NAT TTS model. The model takes both the phonemes (yellow) and the graphemes (pink) of text as input. PnG BERT can be pre-trained in a self-supervised manner on a large text corpus.
  • Figure 2: Augmented PnG NAT model for zero-shot voice cloning. The speaker encoder is separately trained in a speaker verification task and is frozen during TTS training.
  • Figure 3: Naturalness and speaker similarity MOS on CVSS-T breaking down by duration of the translation speech. Bars are the distribution of the durations.