Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment

Yan Gao, Yazheng Yang, Zhibin Lan, Yidong Chen, Min Zhang, Daimeng Wei, Hui Huang, Jinsong Su

TL;DR

This work targets fine-grained code-switch speech translation by integrating a Mixture-of-Experts (MoE) speech projector with language-specific experts and a four-stage training regime to align speech and text across monolingual and CS data. The MoE projector enables language-aware semantic modeling, while the staged transition training uses ASR, monolingual ST, and CS ST data to progressively bridge modalities and data gaps. Empirical results on Fisher and NTUML2021 demonstrate consistent improvements over strong baselines in CS, monolingual, and mixed settings, with ablations confirming the contribution of the MoE design and the proposed losses. The approach advances CS ST by solving semantic alignment and data scarcity challenges, offering a scalable route for cross-lingual speech translation with CS phenomena without excessive manual annotation.

Abstract

Code-switching (CS) speech translation (ST) refers to translating speech that alternates between two or more languages into a target language text, which poses significant challenges due to the complexity of semantic modeling and the scarcity of CS data. Previous studies tend to rely on the model itself to implicitly learn semantic modeling during training, and resort to inefficient and costly manual annotations for these two challenges. To mitigate these limitations, we propose enhancing Large Language Models (LLMs) with a Mixture of Experts (MoE) speech projector, where each expert specializes in the semantic subspace of a specific language, enabling fine-grained modeling of speech features. Additionally, we introduce a multi-stage training paradigm that utilizes readily available monolingual automatic speech recognition (ASR) and monolingual ST data, facilitating speech-text alignment and improving translation capabilities. During training, we leverage a combination of language-specific loss and intra-group load balancing loss to guide the MoE speech projector in efficiently allocating tokens to the appropriate experts, across expert groups and within each group, respectively. To bridge the data gap across different training stages and improve adaptation to the CS scenario, we further employ a transition loss, enabling smooth transitions of data between stages, to effectively address the scarcity of high-quality CS speech translation data. Extensive experiments on widely used datasets demonstrate the effectiveness and generality of our approach.

Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment

TL;DR

This work targets fine-grained code-switch speech translation by integrating a Mixture-of-Experts (MoE) speech projector with language-specific experts and a four-stage training regime to align speech and text across monolingual and CS data. The MoE projector enables language-aware semantic modeling, while the staged transition training uses ASR, monolingual ST, and CS ST data to progressively bridge modalities and data gaps. Empirical results on Fisher and NTUML2021 demonstrate consistent improvements over strong baselines in CS, monolingual, and mixed settings, with ablations confirming the contribution of the MoE design and the proposed losses. The approach advances CS ST by solving semantic alignment and data scarcity challenges, offering a scalable route for cross-lingual speech translation with CS phenomena without excessive manual annotation.

Abstract

Code-switching (CS) speech translation (ST) refers to translating speech that alternates between two or more languages into a target language text, which poses significant challenges due to the complexity of semantic modeling and the scarcity of CS data. Previous studies tend to rely on the model itself to implicitly learn semantic modeling during training, and resort to inefficient and costly manual annotations for these two challenges. To mitigate these limitations, we propose enhancing Large Language Models (LLMs) with a Mixture of Experts (MoE) speech projector, where each expert specializes in the semantic subspace of a specific language, enabling fine-grained modeling of speech features. Additionally, we introduce a multi-stage training paradigm that utilizes readily available monolingual automatic speech recognition (ASR) and monolingual ST data, facilitating speech-text alignment and improving translation capabilities. During training, we leverage a combination of language-specific loss and intra-group load balancing loss to guide the MoE speech projector in efficiently allocating tokens to the appropriate experts, across expert groups and within each group, respectively. To bridge the data gap across different training stages and improve adaptation to the CS scenario, we further employ a transition loss, enabling smooth transitions of data between stages, to effectively address the scarcity of high-quality CS speech translation data. Extensive experiments on widely used datasets demonstrate the effectiveness and generality of our approach.

Paper Structure

This paper contains 27 sections, 14 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Traditional CS Translation
  4. LLMs in CS Translation
  5. Preliminary Study
  6. Does a semantic space gap exist between different languages?
  7. Does sharing the projector across languages affect performance?
  8. Methodology
  9. Model Architecture
  10. Speech Encoder.
  11. MoE Speech Projector.
  12. LLM.
  13. Multi-stage Transition Training
  14. Stage 1.
  15. Stage 2.
  16. ...and 12 more sections

Figures (2)

  • Figure 1: KDE plots of different languages speech representations after using T-SNE.
  • Figure 2: An illustrattion of our proposed model framework. Given that the CS speech involves $m$ different languages, the MoE speech projector assigns one expert group per language, with each group comprising $n$ experts specialized in that language, thus the total number of experts $N = n \times m$. During training, we apply LoRA hu2022lora to fine-tune the parameters of the LLM and the speech encoder, while fully fine-tuning all parameters of the MoE speech projector.