Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradient-Informed Training for Low-Resource Multilingual Speech Translation

Ruiyan Sun, Satoshi Nakamura

Abstract

In low-resource multilingual speech-to-text translation, uniform architectural sharing across languages frequently introduces representation conflicts that impede convergence. This work proposes a principled methodology to automatically determine layer-specific sharing patterns by mining training gradient information. Our approach employs three distinct analysis strategies: distance-based language clustering, self/cross-task divergence metrics for capacity allocation, and joint factorization coupled with canonical correlation analysis for subspace alignment. Extensive evaluation across four language pairs (using the SeamlessM4T-Medium architecture) demonstrates persistent improvements in translation quality metrics.

Gradient-Informed Training for Low-Resource Multilingual Speech Translation

Abstract

In low-resource multilingual speech-to-text translation, uniform architectural sharing across languages frequently introduces representation conflicts that impede convergence. This work proposes a principled methodology to automatically determine layer-specific sharing patterns by mining training gradient information. Our approach employs three distinct analysis strategies: distance-based language clustering, self/cross-task divergence metrics for capacity allocation, and joint factorization coupled with canonical correlation analysis for subspace alignment. Extensive evaluation across four language pairs (using the SeamlessM4T-Medium architecture) demonstrates persistent improvements in translation quality metrics.

Paper Structure

This paper contains 19 sections, 13 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Data
  3. Methodology
  4. Framework Construction
  5. Gradient Analysis Formulations
  6. Method A: Language Grouping via Clustering
  7. Method B: Self versus Cross Gradient Similarity
  8. Method C: Joint SVD and Regularized CCA
  9. Gradient Observations and Analysis Results
  10. GDPS Framework Design and Implementation
  11. Experiments and results
  12. Experimental Setup and Training Configuration
  13. Main Results
  14. Competitive Comparison with IWSLT Systems
  15. Gradient Representation Alignment Analysis
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Overview of the GDPS framework combining the incorporation of gradient-driven decision-making, dynamic parameter configuration, and fine-tuning.
  • Figure 2: Average cross-language similarity per language between GDPS and the SeamlessM4T-Med baseline.