Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pushing the Limits of Zero-shot End-to-End Speech Translation

Ioannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, Marta R. Costa-jussà

TL;DR

ZeroSwot tackles data scarcity and the speech-text modality gap in end-to-end speech translation by training a dedicated speech encoder to align with a massively multilingual MT model through Optimal Transport and CTC-driven subword compression. At inference, the trained speech encoder replaces the MT embedding layer, enabling zero-shot translation from speech to text across all MT languages without any paired ST data. Empirically, ZeroSwot achieves state-of-the-art zero-shot results on MuST-C and strong performance on CoVoST and FLEURS, while offering efficiency advantages over cascaded approaches; ablations highlight the essential role of the speech embedder and the compression module. The work demonstrates substantial multilingual capacity and modality alignment, with limitations including reliance on ASR data and reduced acoustic prosody utilization, pointing to future work in low-resource and speech-to-speech scenarios.

Abstract

Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems, thus hindering their performance. Prior work has attempted to mitigate these challenges by leveraging external MT data and optimizing distance metrics that bring closer the speech-text representations. However, achieving competitive results typically requires some ST data. For this reason, we introduce ZeroSwot, a method for zero-shot ST that bridges the modality gap without any paired ST data. Leveraging a novel CTC compression and Optimal Transport, we train a speech encoder using only ASR data, to align with the representation space of a massively multilingual MT model. The speech encoder seamlessly integrates with the MT model at inference, enabling direct translation from speech to text, across all languages supported by the MT model. Our experiments show that we can effectively close the modality gap without ST data, while our results on MuST-C and CoVoST demonstrate our method's superiority over not only previous zero-shot models, but also supervised ones, achieving state-of-the-art results.

Pushing the Limits of Zero-shot End-to-End Speech Translation

TL;DR

ZeroSwot tackles data scarcity and the speech-text modality gap in end-to-end speech translation by training a dedicated speech encoder to align with a massively multilingual MT model through Optimal Transport and CTC-driven subword compression. At inference, the trained speech encoder replaces the MT embedding layer, enabling zero-shot translation from speech to text across all MT languages without any paired ST data. Empirically, ZeroSwot achieves state-of-the-art zero-shot results on MuST-C and strong performance on CoVoST and FLEURS, while offering efficiency advantages over cascaded approaches; ablations highlight the essential role of the speech embedder and the compression module. The work demonstrates substantial multilingual capacity and modality alignment, with limitations including reliance on ASR data and reduced acoustic prosody utilization, pointing to future work in low-resource and speech-to-speech scenarios.

Abstract

Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems, thus hindering their performance. Prior work has attempted to mitigate these challenges by leveraging external MT data and optimizing distance metrics that bring closer the speech-text representations. However, achieving competitive results typically requires some ST data. For this reason, we introduce ZeroSwot, a method for zero-shot ST that bridges the modality gap without any paired ST data. Leveraging a novel CTC compression and Optimal Transport, we train a speech encoder using only ASR data, to align with the representation space of a massively multilingual MT model. The speech encoder seamlessly integrates with the MT model at inference, enabling direct translation from speech to text, across all languages supported by the MT model. Our experiments show that we can effectively close the modality gap without ST data, while our results on MuST-C and CoVoST demonstrate our method's superiority over not only previous zero-shot models, but also supervised ones, achieving state-of-the-art results.
Paper Structure (36 sections, 11 equations, 3 figures, 17 tables)

This paper contains 36 sections, 11 equations, 3 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Relevant Research
  3. End-to-end Speech Translation
  4. Optimal Transport
  5. Zero-shot Speech Translation
  6. Methodology
  7. Problem Definition & Proposed Solution
  8. Model Architecture
  9. Text Encoder
  10. Acoustic Encoder
  11. Connectionist Temporal Classification
  12. Compression Adapter
  13. Speech Embedder
  14. Semantic Encoder
  15. Optimal Transport
  16. ...and 21 more sections

Figures (3)

  • Figure 1: Proposed Zero-shot Speech Translation
  • Figure 2: Methodology: Speech Encoder Training, Compression Adapter, and Inference with ZeroSwot.
  • Figure 3: Example of the proposed compression.