MEXMA: Token-level objectives improve sentence representations

TL;DR

MEXMA tackles the limitation of cross-lingual sentence encoders trained solely with sentence-level objectives by integrating token-level objectives through cross-unmasking. The method updates both token representations and the sentence embedding via a symmetrical, multilingual architecture and a KoLeo-driven entropy regularization, with a cross-language unmasking objective and an alignment loss enforcing cross-lingual sentence proximity. Empirical results on xsim, xsim++, BUCC, SentEval, and MTEB show state-of-the-art or competitive performance, including strong token alignment across languages and improved downstream task accuracy with smaller models. The work demonstrates that explicit token-level gradients and cross-lingual masking significantly enhance sentence representations, offering practical benefits for multilingual retrieval and understanding, with promising scalability to more languages and modalities.

Abstract

Current pre-trained cross-lingual sentence encoders approaches use sentence-level objectives only. This can lead to loss of information, especially for tokens, which then degrades the sentence representation. We propose MEXMA, a novel approach that integrates both sentence-level and token-level objectives. The sentence representation in one language is used to predict masked tokens in another language, with both the sentence representation and all tokens directly updating the encoder. We show that adding token-level objectives greatly improves the sentence representation quality across several tasks. Our approach outperforms current pre-trained cross-lingual sentence encoders on bi-text mining as well as several downstream tasks. We also analyse the information encoded in our tokens, and how the sentence representation is built from them.

Figures (12)

• Figure 1: MEXMA architecture. Given two translations, we create two views for each, a masked and a clean version (symmetrical architecture), and use the sentence representations from one language to unmask the other (cross-unmasking). We align the clean sentence representations via the alignment loss, and increase the usage of the space with the KoLeo loss.
• Figure 2: Example of LaBSE's token matching. The token in blue is the query token, the tokens in pink are the closest tokens to the query token in the space.
• Figure 3: Example of MEXMA no token-level grad's token matching. The token in blue is the query token, the tokens in pink are the closest tokens to the query token in the space.
• Figure 4: Example of XLM-RoBERTa token matching. The token in blue is the query token, the tokens in pink are the closest tokens to the query token in the space.
• Figure 5: Comparison of SONAR and MEXMA token matching. MEXMA displays the ability to match a token in another sentence in the same language. SONAR matches a translated token. The token in blue is the query token, the tokens in pink are the closest tokens to the query token in the space. MEXMA is on the left, SONAR on the right.