Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages

Midia Yousefi, Yao Qian, Junkun Chen, Gang Wang, Yanqing Liu, Dongmei Wang, Xiaofei Wang, Jian Xue

TL;DR

The proposed Isochrony-Controlled ST controls translation length by predicting the duration of speech and pauses in conjunction with the translation process by providing timing information to the decoder, ensuring it tracks the remaining duration for speech and pauses while generating the translation.

Abstract

End-to-end speech translation (ST), which translates source language speech directly into target language text, has garnered significant attention in recent years. Many ST applications require strict length control to ensure that the translation duration matches the length of the source audio, including both speech and pause segments. Previous methods often controlled the number of words or characters generated by the Machine Translation model to approximate the source sentence's length without considering the isochrony of pauses and speech segments, as duration can vary between languages. To address this, we present improvements to the duration alignment component of our sequence-to-sequence ST model. Our method controls translation length by predicting the duration of speech and pauses in conjunction with the translation process. This is achieved by providing timing information to the decoder, ensuring it tracks the remaining duration for speech and pauses while generating the translation. The evaluation on the Zh-En test set of CoVoST 2, demonstrates that the proposed Isochrony-Controlled ST achieves 0.92 speech overlap and 8.9 BLEU, which has only a 1.4 BLEU drop compared to the ST baseline.

Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages

TL;DR

The proposed Isochrony-Controlled ST controls translation length by predicting the duration of speech and pauses in conjunction with the translation process by providing timing information to the decoder, ensuring it tracks the remaining duration for speech and pauses while generating the translation.

Abstract

End-to-end speech translation (ST), which translates source language speech directly into target language text, has garnered significant attention in recent years. Many ST applications require strict length control to ensure that the translation duration matches the length of the source audio, including both speech and pause segments. Previous methods often controlled the number of words or characters generated by the Machine Translation model to approximate the source sentence's length without considering the isochrony of pauses and speech segments, as duration can vary between languages. To address this, we present improvements to the duration alignment component of our sequence-to-sequence ST model. Our method controls translation length by predicting the duration of speech and pauses in conjunction with the translation process. This is achieved by providing timing information to the decoder, ensuring it tracks the remaining duration for speech and pauses while generating the translation. The evaluation on the Zh-En test set of CoVoST 2, demonstrates that the proposed Isochrony-Controlled ST achieves 0.92 speech overlap and 8.9 BLEU, which has only a 1.4 BLEU drop compared to the ST baseline.

Paper Structure

This paper contains 7 sections, 6 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. End-to-End Speech-to-Text Translation
  3. Training
  4. Inference
  5. Experiments
  6. Results and analysis
  7. Conclusions

Figures (3)

  • Figure 1: The proposed Isochrony-controlled E2E Speech-to-Text Translation model.
  • Figure 2: Length histogram of the fine-tuning data.
  • Figure 3: Decoding examples of the proposed Isochrony-Controlled ST and Baseline Translation. The first column shows the provided Chinese utterance transcription. The second column is the ground truth translation, and the third and fourth columns are the generated translations by the Isochrony-Controlled ST and Baseline Translation, respectively. Please note that the BLEU score for both the Isochrony-Controlled and baseline ST models range from 8.9 to 10.3, therefore, there are errors in the Translations in both scenarios.