EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning

TL;DR

EMS tackles the challenge of training efficient massively multilingual sentence embeddings by coupling cross-lingual token-level reconstruction (XTR) with sentence-level contrastive learning in a dual-encoder setup. The approach uses a language-embedding layer and lightweight MLPs to produce language-agnostic representations across 62 languages with substantially less parallel data and GPU time than prior models. Empirical results show EMS achieving comparable or better cross-lingual retrieval and zero-shot classification performance across Tatoeba, Flores, BUCC, ParaCrawl, MLDoc, and sentiment tasks, with strong data-efficiency and faster training. The work demonstrates that combining XTR and a carefully designed contrastive objective yields robust multilingual embeddings, and it highlights practical implications for deploying multilingual NLP with limited resources.

Abstract

Massively multilingual sentence representation models, e.g., LASER, SBERT-distill, and LaBSE, help significantly improve cross-lingual downstream tasks. However, the use of a large amount of data or inefficient model architectures results in heavy computation to train a new model according to our preferred languages and domains. To resolve this issue, we introduce efficient and effective massively multilingual sentence embedding (EMS), using cross-lingual token-level reconstruction (XTR) and sentence-level contrastive learning as training objectives. Compared with related studies, the proposed model can be efficiently trained using significantly fewer parallel sentences and GPU computation resources. Empirical results showed that the proposed model significantly yields better or comparable results with regard to cross-lingual sentence retrieval, zero-shot cross-lingual genre classification, and sentiment classification. Ablative analyses demonstrated the efficiency and effectiveness of each component of the proposed model. We release the codes for model training and the EMS pre-trained sentence embedding model, which supports 62 languages ( https://github.com/Mao-KU/EMS ).

Figures (1)

• Figure 1: Training architecture of EMS.$\mathbf{u}$ and $\mathbf{v}$ are MSEs for inference, and the model components in the red dashed rectangle are used for inference. $\mathbf{u}_{la}$ and $\mathbf{v}_{la}$ are the target language embeddings. $\oplus$ denotes the hidden vector concatenation. The training data batch sample is given in the blue dashed box. Orange arrows and dashed box denote the gold token distributions within the generative objective, specifically the discrete uniform distributions for the tokens in $S_{en}$ and $S_{fr}$, denoted by $\mathlarger{p}_{S_{en}}$ and $\mathlarger{p}_{S_{fr}}$, respectively. The part within the red dashed box indicates the pre-trained EMS model for downstream tasks.