Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Listen, Attend, Understand: a Regularization Technique for Stable E2E Speech Translation Training on High Variance labels

Yacouba Diarra, Michael Leventhal

TL;DR

This work tackles instability in end-to-end speech translation when target labels are highly variable by introducing LAU, a semantic regularization that steers the acoustic encoder toward a frozen embedding space via an auxiliary loss. The approach combines a standard sequence objective with a semantic head that aligns encoder outputs to pretrained text embeddings, controlled by a weight $\lambda$, and preserves inference cost. On a Bambara→French 30-hour dataset, LAU yields performance close to a fully pre-trained E2E-ST and outperforms a cascaded ASR→MT pipeline, while also enhancing semantic preservation for downstream SLU tasks. A novel Total Parameter Drift metric demonstrates that semantic constraints actively reorganize encoder weights to prioritize meaning over literal phonetics, underscoring LAU as a robust, data-efficient training regularizer for low-resource speech translation.

Abstract

End-to-End Speech Translation often shows slower convergence and worse performance when target transcriptions exhibit high variance and semantic ambiguity. We propose Listen, Attend, Understand (LAU), a semantic regularization technique that constrains the acoustic encoder's latent space during training. By leveraging frozen text embeddings to provide a directional auxiliary loss, LAU injects linguistic groundedness into the acoustic representation without increasing inference cost. We evaluate our method on a Bambara-to-French dataset with 30 hours of Bambara speech translated by non-professionals. Experimental results demonstrate that LAU models achieve comparable performance by standard metrics compared to an E2E-ST system pretrained with 100\% more data and while performing better in preserving semantic meaning. Furthermore, we introduce Total Parameter Drift as a metric to quantify the structural impact of regularization to demonstrate that semantic constraints actively reorganize the encoder's weights to prioritize meaning over literal phonetics. Our findings suggest that LAU is a robust alternative to post-hoc rescoring and a valuable addition to E2E-ST training, especially when training data is scarce and/or noisy.

Listen, Attend, Understand: a Regularization Technique for Stable E2E Speech Translation Training on High Variance labels

TL;DR

This work tackles instability in end-to-end speech translation when target labels are highly variable by introducing LAU, a semantic regularization that steers the acoustic encoder toward a frozen embedding space via an auxiliary loss. The approach combines a standard sequence objective with a semantic head that aligns encoder outputs to pretrained text embeddings, controlled by a weight , and preserves inference cost. On a Bambara→French 30-hour dataset, LAU yields performance close to a fully pre-trained E2E-ST and outperforms a cascaded ASR→MT pipeline, while also enhancing semantic preservation for downstream SLU tasks. A novel Total Parameter Drift metric demonstrates that semantic constraints actively reorganize encoder weights to prioritize meaning over literal phonetics, underscoring LAU as a robust, data-efficient training regularizer for low-resource speech translation.

Abstract

End-to-End Speech Translation often shows slower convergence and worse performance when target transcriptions exhibit high variance and semantic ambiguity. We propose Listen, Attend, Understand (LAU), a semantic regularization technique that constrains the acoustic encoder's latent space during training. By leveraging frozen text embeddings to provide a directional auxiliary loss, LAU injects linguistic groundedness into the acoustic representation without increasing inference cost. We evaluate our method on a Bambara-to-French dataset with 30 hours of Bambara speech translated by non-professionals. Experimental results demonstrate that LAU models achieve comparable performance by standard metrics compared to an E2E-ST system pretrained with 100\% more data and while performing better in preserving semantic meaning. Furthermore, we introduce Total Parameter Drift as a metric to quantify the structural impact of regularization to demonstrate that semantic constraints actively reorganize the encoder's weights to prioritize meaning over literal phonetics. Our findings suggest that LAU is a robust alternative to post-hoc rescoring and a valuable addition to E2E-ST training, especially when training data is scarce and/or noisy.
Paper Structure (17 sections, 9 equations, 3 figures, 2 tables)

This paper contains 17 sections, 9 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. End-to-End Speech Translation
  4. Introducing Semantics in Spoken Language Understanding
  5. Implementation & Experiments
  6. Sequence Loss.
  7. Semantic Loss.
  8. Results
  9. LAU for Regularization
  10. BLEU & WER evaluation
  11. LAU Potential for SLU
  12. Cluster purity.
  13. Normalized Mutual Information (NMI).
  14. Discussion
  15. High Potential for LRL Speech Applications
  16. ...and 2 more sections

Figures (3)

  • Figure 1: a) The training graph adds a shallow semantic head to the encoder output to align it with frozen reference embeddings (LAU regularization). b) During inference, the semantic branch is dropped, resulting in a standard Encoder-Decoder architecture.
  • Figure 2: Total parameter drift for different semantic regularization loss choices and weightings
  • Figure 3: Multitask conflict shows the new regularization technique is active as higher values for $\lambda$ drive more encoder updates to align with the semantic constraint to the detriment of lexical accuracy