Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech

Abstract

Cross-lingual sentence encoders typically cover only a few hundred languages and often trade downstream quality for stronger alignment, limiting their adoption. We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-modal sentence embedding models that natively embed text, speech, code, and mathematical expressions in a single semantic space, while delivering state-of-the-art downstream performance at the scale of thousands of languages, from high-resource to extremely low-resource varieties. To reach this scale without representation collapse, we use progressive training. We first learn a strong foundational space for 200 languages with an LLM-initialized encoder-decoder, combining token-level decoding with a novel split-softmax contrastive loss and synthetic hard negatives. Building on this foundation, we expand to several thousands language varieties via a two-stage teacher-student encoder distillation framework. Finally, we demonstrate the cross-modal extensibility of this space by seamlessly mapping 177 spoken languages into it. OmniSONAR halves cross-lingual similarity search error on the 200-language FLORES dataset and reduces error by a factor of 15 on the 1,560-language BIBLE benchmark. It also enables strong translation, outperforming NLLB-3B on multilingual benchmarks and exceeding prior models (including much larger LLMs) by 15 chrF++ points on 1,560 languages into English BIBLE translation. OmniSONAR also performs strongly on MTEB and XLCoST. For speech, OmniSONAR achieves a 43% lower similarity-search error and reaches 97% of SeamlessM4T speech-to-text quality, despite being zero-shot for translation (trained only on ASR data). Finally, by training an encoder-decoder LM, Spectrum, exclusively on English text processing OmniSONAR embedding sequences, we unlock high-performance transfer to thousands of languages and speech for complex downstream tasks.

Figures (18)

• Figure 1: The OmniSONAR training stages. In Stage 1, we train our LLM-initialized encoder-decoder on translation data with a decoding loss. In Stage 2, we introduce an encoder bottleneck via pooling and train with a combination of contrastive and decoding objectives. In Stage 3, we introduce hard negatives and continue training with a split-softmax contrastive objective and the decoding loss. In Stage 4, we extend the space to omnilingual-level language coverage by training with teacher-student distillation on 4,200 language varieties with a combination of MSE and Contrastive objectives, while first warming-up the omnilingual tokenization with MSE-based distillation. Lastly, in Stage 5, we extend the omnilingual space to the speech modality with teacher-student distillation using ASR data.
• Figure 2: Example of OmniSONAR natural text prompting on both encoder and decoder sides. Source sentences, which are input to the encoder, are prefixed with language identifiers. The format used is "[language name]:". Starting from the omnilingual extension, this prefix can be randomly replaced by "Unspecified language:". Target sentences, provided to the decoder, include task specifications, output language information, and information about data provenance (indicating whether the translation is human-labeled, automatically mined, or back-translated), following nllb. Specifically, we use the prompts such as This is a possible translation in [language name]: for translation tasks and This is a possible natural language explanation in English: for code and math explanation tasks. This input formatting is applied consistently across all training stages. We provide the full list of prompts in Appendix \ref{['tab:appendix/prompts']}.
• Figure 3: Loss function parameters for new languages. Results on Bible dev. chrF++ obtained with the OmniSONAR-200 decoder. Vertical dashed lines indicate the final configuration used for OmniSONAR.
• Figure 4: Cross-lingual similarity (xsim) and translation quality (chrF++) for encoders trained on progressively more language groups, evaluated on BIBLE dev X-Eng. For translation we use the OmniSONAR-200 decoder. In the Top subfigures (a) and (b), we display Resource-based grouping: Groups A-G sorted by resourcefulness. In the Bottom subfigures (c) and (d), we display Family-based grouping: Group A = Indo-European (largest family) and Groups B-F = other families. In bold is the best performance per column (excluding OmniSONAR). The "All" column is the average across all 1,560 languages. OmniSONAR is trained on all groups plus an additional group containing 1,864 extremely low resource languages.
• Figure 5: Architecture of the Spectrum model. An encoder processes a sequence of sentence-level OmniSONAR embeddings, while the decoder is a token-level transformer that generates answers auto-regressively from a prefixed prompt. The decoder achieves this by performing cross-attention over the outputs of the encoder. We refer to the encoder as the "SONAR tower" and the decoder as the "token tower".