Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coupling Speech Encoders with Downstream Text Models

Ciprian Chelba, Johan Schalkwyk

TL;DR

The paper addresses data scarcity in end-to-end automatic speech translation by proposing a modular cascade architecture that preserves state-of-the-art ASR and MT performance. It introduces an exporter layer trained with $L_2$-loss to align ASR encoder outputs with MT input embeddings for the 1-best sequence, enabling gradient flow from the MT model back into the ASR module and guaranteeing no worse performance than the 1-best cascade. Experiments on the En-De MuST-C v2 corpus show that matched-embeddings cascades can improve AST when using a WMT MT backend, though gains are more modest with MuST-C back-ends due to task adaptation. The approach offers a general framework for coupling ASR encoders with immutable text models, including large language models, and demonstrates how speech, transcription, and translated transcription data can be leveraged to boost downstream translation quality.

Abstract

We present a modular approach to building cascade speech translation (AST) models that guarantees that the resulting model performs no worse than the 1-best cascade baseline while preserving state-of-the-art speech recognition (ASR) and text translation (MT) performance for a given task. Our novel contribution is the use of an ``exporter'' layer that is trained under L2-loss to ensure a strong match between ASR embeddings and the MT token embeddings for the 1-best sequence. The ``exporter'' output embeddings are fed directly to the MT model in lieu of 1-best token embeddings, thus guaranteeing that the resulting model performs no worse than the 1-best cascade baseline, while allowing back-propagation gradient to flow from the MT model into the ASR components. The matched-embeddings cascade architecture provide a significant improvement over its 1-best counterpart in scenarios where incremental training of the MT model is not an option and yet we seek to improve quality by leveraging (speech, transcription, translated transcription) data provided with the AST task. The gain disappears when the MT model is incrementally trained on the parallel text data available with the AST task. The approach holds promise for other scenarios that seek to couple ASR encoders and immutable text models, such at large language models (LLM).

Coupling Speech Encoders with Downstream Text Models

TL;DR

The paper addresses data scarcity in end-to-end automatic speech translation by proposing a modular cascade architecture that preserves state-of-the-art ASR and MT performance. It introduces an exporter layer trained with -loss to align ASR encoder outputs with MT input embeddings for the 1-best sequence, enabling gradient flow from the MT model back into the ASR module and guaranteeing no worse performance than the 1-best cascade. Experiments on the En-De MuST-C v2 corpus show that matched-embeddings cascades can improve AST when using a WMT MT backend, though gains are more modest with MuST-C back-ends due to task adaptation. The approach offers a general framework for coupling ASR encoders with immutable text models, including large language models, and demonstrates how speech, transcription, and translated transcription data can be leveraged to boost downstream translation quality.

Abstract

We present a modular approach to building cascade speech translation (AST) models that guarantees that the resulting model performs no worse than the 1-best cascade baseline while preserving state-of-the-art speech recognition (ASR) and text translation (MT) performance for a given task. Our novel contribution is the use of an ``exporter'' layer that is trained under L2-loss to ensure a strong match between ASR embeddings and the MT token embeddings for the 1-best sequence. The ``exporter'' output embeddings are fed directly to the MT model in lieu of 1-best token embeddings, thus guaranteeing that the resulting model performs no worse than the 1-best cascade baseline, while allowing back-propagation gradient to flow from the MT model into the ASR components. The matched-embeddings cascade architecture provide a significant improvement over its 1-best counterpart in scenarios where incremental training of the MT model is not an option and yet we seek to improve quality by leveraging (speech, transcription, translated transcription) data provided with the AST task. The gain disappears when the MT model is incrementally trained on the parallel text data available with the AST task. The approach holds promise for other scenarios that seek to couple ASR encoders and immutable text models, such at large language models (LLM).
Paper Structure (11 sections, 2 figures, 7 tables)

This paper contains 11 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Modeling
  3. L2 Matcher for Cascade AST
  4. Experiments
  5. ASR experiments
  6. MT experiments
  7. AST Experiments
  8. L2 Matcher Experiments
  9. Related Work
  10. Conclusions and Future Work
  11. Acknowledgements

Figures (2)

  • Figure 1: Graph model description for training the L2 loss matcher.
  • Figure 2: Graph model description for using the L2 loss matcher in cascade AST.