POTSA: A Cross-Lingual Speech Alignment Framework for Low Resource Speech-to-Text Translation

TL;DR

The paper tackles cross-lingual representation bias in SpeechLLMs for speech-to-text translation (S2TT). It introduces POTSA, a cross-lingual speech alignment framework that uses cross-lingual parallel speech pairs and Optimal Transport to align source speech representations, comprising a Bias Compensation module, token-wise OT constraints on a Q-Former, and an online layer scheduling strategy while keeping the encoder and LLM frozen. The approach achieves state-of-the-art BLEU results on FLEURS, including +0.93 average BLEU across five languages and +5.05 BLEU on zero-shot languages with only 10 hours of parallel speech per source language, demonstrating strong gains with limited data. This cross-llingual alignment enables better transfer to low-resource languages and can be integrated into existing SpeechLLMs to improve multilingual S2TT performance in practical settings.

Abstract

Speech Large Language Models (SpeechLLMs) have achieved breakthroughs in multilingual speech-to-text translation (S2TT). However, existing approaches often overlook semantic commonalities across source languages, leading to biased translation performance. In this work, we propose \textbf{POTSA} (Parallel Optimal Transport for Speech Alignment), a new framework based on cross-lingual parallel speech pairs and Optimal Transport (OT), designed to bridge high- and low-resource translation gaps. First, we introduce a Bias Compensation module to coarsely align initial speech representations across languages. Second, we impose token-level OT constraints on a Q-Former using parallel speech pairs to establish fine-grained consistency of representations. Then, we apply a layer scheduling strategy to focus OT constraints on the most semantically beneficial layers. Experiments on the FLEURS dataset show that our method achieves SOTA performance, with +0.93 BLEU on average over five common languages and +5.05 BLEU on zero-shot languages, using only 10 hours of parallel speech per source language.

Figures (3)

• Figure 1: (a): Language-specific clusters before alignment. (b): Utterances with the identical semantics converge toward a shared representation (pivot language).
• Figure 2: Model Architecture of Our Cross-Lingual Speech Alignment Framework
• Figure 3: Alignment effectiveness of our framework