SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation

TL;DR

SAMU-XLSR introduces an utterance-level, multimodal cross-lingual speech representation by distilling LaBSE's semantically aligned sentence embeddings into a pre-trained XLS-R speech encoder. The framework uses Self-Attention pooling and a cosine-distance objective to learn embeddings that align speech, text, and translations across 109 written languages and 22 spoken languages, trained on 6.8k hours from CommonVoice with data-balancing to address resource disparities. It demonstrates zero-shot cross-lingual speech-to-text and speech-to-speech translation retrieval across multiple datasets, and maintains competitive performance on phoneme recognition and ASR, illustrating broader applicability beyond retrieval. The work highlights design choices (loss/pooling, data balancing) and shows potential for large-scale data mining to build parallel speech-text resources for speech translation tasks.

Abstract

We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation learning framework. Unlike previous works on speech representation learning, which learns multilingual contextual speech embedding at the resolution of an acoustic frame (10-20ms), this work focuses on learning multimodal (speech-text) multilingual speech embedding at the resolution of a sentence (5-10s) such that the embedding vector space is semantically aligned across different languages. We combine state-of-the-art multilingual acoustic frame-level speech representation learning model XLS-R with the Language Agnostic BERT Sentence Embedding (LaBSE) model to create an utterance-level multimodal multilingual speech encoder SAMU-XLSR. Although we train SAMU-XLSR with only multilingual transcribed speech data, cross-lingual speech-text and speech-speech associations emerge in its learned representation space. To substantiate our claims, we use SAMU-XLSR speech encoder in combination with a pre-trained LaBSE text sentence encoder for cross-lingual speech-to-text translation retrieval, and SAMU-XLSR alone for cross-lingual speech-to-speech translation retrieval. We highlight these applications by performing several cross-lingual text and speech translation retrieval tasks across several datasets.

Figures (5)

• Figure 1: An illustration of the cross-lingual multimodal embedding space.
• Figure 2: A pedagogical description of how learning with transcribed speech data using $\tt LaBSE$ as the teacher could lead to the emergence of cross-lingual speech and text associations. In this illustration, we use English speech $x^{(\text{EN})}$ and its transcription $y^{(\text{EN})}$ for training. $\tt SAMU\text{-}XLSR$'s parameters are tuned to close the distance between the speech embedding given by $\tt SAMU\text{-}XLSR$ in orange and $\tt LaBSE$'s embedding (Anchor) of the corresponding text transcript in green. Since $\tt LaBSE$'s text embedding space is semantically-aligned across various languages, by pulling the speech embedding towards the anchor embedding, we automatically learn cross-lingual speech-text alignments without ever seeing cross-lingual associations during training. In practice, we train $\tt SAMU\text{-}XLSR$ with multilingual transcribed speech, not just English.
• Figure 3: An illustration of the multimodal training framework
• Figure 4: Re-balancing the training set with different values of the smoothing parameter $\alpha$
• Figure 5: We extract the representation sequence from a Pre-trained $\tt SAMU\text{-}XLSR$ (our proposed model) from before the attention pooling layer. Next, we compute the cosine similarity between the adjacent feature vectors to compute a sequence of distances and use a peak finding algorithm to detect the local peaks. After tuning the peak threshold in the peak finding algorithm, we observe that the peaks correspond to the underlying word boundaries.