Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?

Ioannis Tsiamas, Matthias Sperber, Andrew Finch, Sarthak Garg

TL;DR

An evaluation methodology and a focused benchmark aimed at capturing a wide range of prosodic phenomena are introduced and found that S2TT models possess some internal representation of prosody, but the prosody signal is often not strong enough to affect the translations.

Abstract

The prosody of a spoken utterance, including features like stress, intonation and rhythm, can significantly affect the underlying semantics, and as a consequence can also affect its textual translation. Nevertheless, prosody is rarely studied within the context of speech-to-text translation (S2TT) systems. In particular, end-to-end (E2E) systems have been proposed as well-suited for prosody-aware translation because they have direct access to the speech signal when making translation decisions, but the understanding of whether this is successful in practice is still limited. A main challenge is the difficulty of evaluating prosody awareness in translation. To address this challenge, we introduce an evaluation methodology and a focused benchmark (named ContraProST) aimed at capturing a wide range of prosodic phenomena. Our methodology uses large language models and controllable text-to-speech (TTS) to generate contrastive examples. Through experiments in translating English speech into German, Spanish, and Japanese, we find that (a) S2TT models possess some internal representation of prosody, but the prosody signal is often not strong enough to affect the translations, (b) E2E systems outperform cascades of speech recognition and text translation systems, confirming their theoretical advantage in this regard, and (c) certain cascaded systems also capture prosodic information in the translation, but only to a lesser extent that depends on the particulars of the transcript's surface form.

Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?

TL;DR

An evaluation methodology and a focused benchmark aimed at capturing a wide range of prosodic phenomena are introduced and found that S2TT models possess some internal representation of prosody, but the prosody signal is often not strong enough to affect the translations.

Abstract

The prosody of a spoken utterance, including features like stress, intonation and rhythm, can significantly affect the underlying semantics, and as a consequence can also affect its textual translation. Nevertheless, prosody is rarely studied within the context of speech-to-text translation (S2TT) systems. In particular, end-to-end (E2E) systems have been proposed as well-suited for prosody-aware translation because they have direct access to the speech signal when making translation decisions, but the understanding of whether this is successful in practice is still limited. A main challenge is the difficulty of evaluating prosody awareness in translation. To address this challenge, we introduce an evaluation methodology and a focused benchmark (named ContraProST) aimed at capturing a wide range of prosodic phenomena. Our methodology uses large language models and controllable text-to-speech (TTS) to generate contrastive examples. Through experiments in translating English speech into German, Spanish, and Japanese, we find that (a) S2TT models possess some internal representation of prosody, but the prosody signal is often not strong enough to affect the translations, (b) E2E systems outperform cascades of speech recognition and text translation systems, confirming their theoretical advantage in this regard, and (c) certain cascaded systems also capture prosodic information in the translation, but only to a lesser extent that depends on the particulars of the transcript's surface form.

Paper Structure

This paper contains 26 sections, 11 equations, 7 figures, 11 tables.

Table of Contents

  1. Introduction
  2. The ContraProST Benchmark
  3. Categorization of Prosodic Phenomena
  4. Prosodic Example Generation
  5. Oracle Translation
  6. Controllable Speech Synthesis
  7. Contrastive Evaluation
  8. Contrastive Likelihood
  9. Contrastive Translation Quality
  10. Experimental Setup
  11. Data Generation
  12. Speech-to-text Translation Models
  13. Metrics
  14. Experimental Results
  15. Related Work
  16. ...and 11 more sections

Figures (7)

  • Figure 1: The Data Generation process for ContraProST.
  • Figure 2: Regression Analysis of model types and model sizes per language pair.
  • Figure 3: Upper: Model performance per category (En-De). Lower Model performance comparisons (En-De), (a): SALMONN-13B vs. SeamlessM4T-v2-Large, (b) SeamlessM4T-v2-Large(E2E) vs. SeamlessM4T-v2-Large(cascade), (c) SeamlessM4T-v2-Large(cascade) vs. Whisper-v3-Large/NLLB-3.3B.
  • Figure 4: Regression Analysis of language pairs.
  • Figure 5: Correlation Matrix of the metrics across all language pairs and models.
  • ...and 2 more figures