Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aligning Pre-trained Models for Spoken Language Translation

Šimon Sedláček, Santosh Kesiraju, Alexander Polok, Jan Černocký

TL;DR

This paper investigates a novel approach to end-to-end speech translation (ST) based on aligning frozen pre-trained automatic speech recognition and machine translation models via a small connector module (Q-Former, the authors' Subsampler-Transformer Encoder), and finds that the connectors can serve as domain adapters for the foundation MT models.

Abstract

This paper investigates a novel approach to end-to-end speech translation (ST) based on aligning frozen pre-trained automatic speech recognition (ASR) and machine translation (MT) models via a small connector module (Q-Former, our Subsampler-Transformer Encoder). This connector bridges the gap between the speech and text modalities, transforming ASR encoder embeddings into the latent representation space of the MT encoder while being the only part of the system optimized during training. Experiments are conducted on the How2 English-Portuguese dataset as we investigate the alignment approach in a small-scale scenario focusing on ST. While keeping the size of the connector module constant and small in comparison ( < 5% of the size of the larger aligned models), increasing the size and capability of the foundation ASR and MT models universally improves translation results. We also find that the connectors can serve as domain adapters for the foundation MT models, significantly improving translation performance in the aligned ST setting. We conclude that this approach represents a viable and scalable approach to training end-to-end ST systems.

Aligning Pre-trained Models for Spoken Language Translation

TL;DR

This paper investigates a novel approach to end-to-end speech translation (ST) based on aligning frozen pre-trained automatic speech recognition and machine translation models via a small connector module (Q-Former, the authors' Subsampler-Transformer Encoder), and finds that the connectors can serve as domain adapters for the foundation MT models.

Abstract

This paper investigates a novel approach to end-to-end speech translation (ST) based on aligning frozen pre-trained automatic speech recognition (ASR) and machine translation (MT) models via a small connector module (Q-Former, our Subsampler-Transformer Encoder). This connector bridges the gap between the speech and text modalities, transforming ASR encoder embeddings into the latent representation space of the MT encoder while being the only part of the system optimized during training. Experiments are conducted on the How2 English-Portuguese dataset as we investigate the alignment approach in a small-scale scenario focusing on ST. While keeping the size of the connector module constant and small in comparison ( < 5% of the size of the larger aligned models), increasing the size and capability of the foundation ASR and MT models universally improves translation results. We also find that the connectors can serve as domain adapters for the foundation MT models, significantly improving translation performance in the aligned ST setting. We conclude that this approach represents a viable and scalable approach to training end-to-end ST systems.

Paper Structure

This paper contains 26 sections, 1 equation, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Findings and contributions
  4. Alignment architectures
  5. ECD
  6. ECED
  7. Connector modules
  8. Q-Former
  9. STE connector
  10. Experimental setup
  11. Data and evaluation
  12. Foundation and baseline models
  13. ASR models
  14. MT models
  15. ST baselines
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Diagram of the ECD (A) and the ECED (B) alignment architectures for ST. Modules annotated with the '*' symbol are frozen. In the ECD scenario, the connector outputs are passed directly to the cross-attention (C/A) connection of the MT decoder. For the ECED architecture, the connector outputs are directly injected past the input embedding layer of the MT encoder. Additional task prompt word embeddings can be prepended to the connector outputs before entering the MT encoder.
  • Figure 2: Diagram of the two connector modules. Variant A) is the Q-Former, variant B) is the STE connector.
  • Figure 3: Performance study of the STE connector and the Q-Former (with varying numbers of queries) in relation to input utterance lengths. The final BLEU score is obtained by averaging BLEUs computed for both the val and testHow2 sets.