Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SENSE models: an open source solution for multilingual and multimodal semantic-based tasks

Salima Mdhaffar, Haroun Elleuch, Chaimae Chellaf, Ha Nguyen, Yannick Estève

TL;DR

SENSE presents an open-source, teacher–student framework for multilingual and multimodal semantic representations, building on SAMU-XLSR and SONAR by using a stronger text teacher (BGE-M3) and a better speech encoder (w2v-BERT 2.0). Trained on Common Voice across 83 languages and integrated into SpeechBrain, SENSE achieves competitive results across retrieval, SLU, summarization, and translation, and provides novel analyses of frame-level attention to reveal where semantic content concentrates in speech. The work delivers a practical, extensible resource for cross-lingual speech-text alignment and highlights non-uniform distribution of semantic information across utterances, encouraging further research into semantic embedding dynamics in multilingual contexts.

Abstract

This paper introduces SENSE (Shared Embedding for N-lingual Speech and tExt), an open-source solution inspired by the SAMU-XLSR framework and conceptually similar to Meta AI's SONAR models. These approaches rely on a teacher-student framework to align a self-supervised speech encoder with the language-agnostic continuous representations of a text encoder at the utterance level. We describe how the original SAMU-XLSR method has been updated by selecting a stronger teacher text model and a better initial speech encoder. The source code for training and using SENSE models has been integrated into the SpeechBrain toolkit, and the first SENSE model we trained has been publicly released. We report experimental results on multilingual and multimodal semantic tasks, where our SENSE model achieves highly competitive performance. Finally, this study offers new insights into how semantics are captured in such semantically aligned speech encoders.

SENSE models: an open source solution for multilingual and multimodal semantic-based tasks

TL;DR

SENSE presents an open-source, teacher–student framework for multilingual and multimodal semantic representations, building on SAMU-XLSR and SONAR by using a stronger text teacher (BGE-M3) and a better speech encoder (w2v-BERT 2.0). Trained on Common Voice across 83 languages and integrated into SpeechBrain, SENSE achieves competitive results across retrieval, SLU, summarization, and translation, and provides novel analyses of frame-level attention to reveal where semantic content concentrates in speech. The work delivers a practical, extensible resource for cross-lingual speech-text alignment and highlights non-uniform distribution of semantic information across utterances, encouraging further research into semantic embedding dynamics in multilingual contexts.

Abstract

This paper introduces SENSE (Shared Embedding for N-lingual Speech and tExt), an open-source solution inspired by the SAMU-XLSR framework and conceptually similar to Meta AI's SONAR models. These approaches rely on a teacher-student framework to align a self-supervised speech encoder with the language-agnostic continuous representations of a text encoder at the utterance level. We describe how the original SAMU-XLSR method has been updated by selecting a stronger teacher text model and a better initial speech encoder. The source code for training and using SENSE models has been integrated into the SpeechBrain toolkit, and the first SENSE model we trained has been publicly released. We report experimental results on multilingual and multimodal semantic tasks, where our SENSE model achieves highly competitive performance. Finally, this study offers new insights into how semantics are captured in such semantically aligned speech encoders.

Paper Structure

This paper contains 37 sections, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. SAMU-XSLR framework
  3. The first SENSE model
  4. Teacher model selection
  5. SSL speech encoder selection
  6. SENSE training details
  7. Experimental Motivations
  8. Cross‐lingual and Multimodal Retrieval
  9. Spoken Language Understanding and Summarization
  10. Speech‐to‐Text Translation
  11. Datasets and experimental framework
  12. Multilingual and Multimodal Translation Retrieval task
  13. X Speech $\to$ EN Speech retrieval
  14. EN Speech $\to$ Y Speech retrieval
  15. X Speech $\to$ Y Speech retrieval
  16. ...and 22 more sections

Figures (4)

  • Figure 1: SAMU-XLSR training framework
  • Figure 2: Average Attention Logit (Normalized Position)
  • Figure 3: Example of attention‐weight distribution with VAD information and forced word alignment in French.
  • Figure 4: Example of attention‐weight distribution with VAD information and forced word alignment in German (spoken translation from French)