Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding

Kaiyan Zhao, Qiyu Wu, Xin-Qiang Cai, Yoshimasa Tsuruoka

TL;DR

This work tackles the challenge of learning high-quality cross-lingual sentence embeddings by enhancing contrastive learning with multiple positives drawn from multilingual translations. The authors propose MPCL, a framework that constructs a set of positives for each anchor sentence and optimizes a multi-positive contrastive loss to capture both pairwise similarity and transitive relationships across languages. Empirical results on diverse backbones (including LaBSE, XLM-R, and mBERT variants) show that MPCL improves retrieval, semantic similarity, and classification metrics, and yields better cross-lingual transfer to unseen languages, sometimes surpassing state-of-the-art single-positive methods. The findings indicate that leveraging multilingual positives enriches the embedding space and offers robustness across tasks, with potential for broader language coverage and refined positive weighting in future work.

Abstract

Learning multi-lingual sentence embeddings is a fundamental task in natural language processing. Recent trends in learning both mono-lingual and multi-lingual sentence embeddings are mainly based on contrastive learning (CL) among an anchor, one positive, and multiple negative instances. In this work, we argue that leveraging multiple positives should be considered for multi-lingual sentence embeddings because (1) positives in a diverse set of languages can benefit cross-lingual learning, and (2) transitive similarity across multiple positives can provide reliable structural information for learning. In order to investigate the impact of multiple positives in CL, we propose a novel approach, named MPCL, to effectively utilize multiple positive instances to improve the learning of multi-lingual sentence embeddings. Experimental results on various backbone models and downstream tasks demonstrate that MPCL leads to better retrieval, semantic similarity, and classification performances compared to conventional CL. We also observe that in unseen languages, sentence embedding models trained on multiple positives show better cross-lingual transfer performance than models trained on a single positive instance.

Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding

TL;DR

This work tackles the challenge of learning high-quality cross-lingual sentence embeddings by enhancing contrastive learning with multiple positives drawn from multilingual translations. The authors propose MPCL, a framework that constructs a set of positives for each anchor sentence and optimizes a multi-positive contrastive loss to capture both pairwise similarity and transitive relationships across languages. Empirical results on diverse backbones (including LaBSE, XLM-R, and mBERT variants) show that MPCL improves retrieval, semantic similarity, and classification metrics, and yields better cross-lingual transfer to unseen languages, sometimes surpassing state-of-the-art single-positive methods. The findings indicate that leveraging multilingual positives enriches the embedding space and offers robustness across tasks, with potential for broader language coverage and refined positive weighting in future work.

Abstract

Learning multi-lingual sentence embeddings is a fundamental task in natural language processing. Recent trends in learning both mono-lingual and multi-lingual sentence embeddings are mainly based on contrastive learning (CL) among an anchor, one positive, and multiple negative instances. In this work, we argue that leveraging multiple positives should be considered for multi-lingual sentence embeddings because (1) positives in a diverse set of languages can benefit cross-lingual learning, and (2) transitive similarity across multiple positives can provide reliable structural information for learning. In order to investigate the impact of multiple positives in CL, we propose a novel approach, named MPCL, to effectively utilize multiple positive instances to improve the learning of multi-lingual sentence embeddings. Experimental results on various backbone models and downstream tasks demonstrate that MPCL leads to better retrieval, semantic similarity, and classification performances compared to conventional CL. We also observe that in unseen languages, sentence embedding models trained on multiple positives show better cross-lingual transfer performance than models trained on a single positive instance.
Paper Structure (29 sections, 2 equations, 5 figures, 8 tables)

This paper contains 29 sections, 2 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Multiple Positives in Contrastive Learning for Multi-lingual Sentence Embeddings
  3. Contrastive learning for sentence embeddings
  4. Multiple Positives in Contrastive Learning
  5. Dataset Construction in MPCL
  6. Experiments
  7. Details of Training Dataset
  8. Baselines
  9. Training Details
  10. Evaluation Tasks
  11. Bitext Mining
  12. Semantic Similarity
  13. Classification
  14. Main Experimental Results
  15. Multiple Positives Yield Better Performance than Single Positive
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Different shapes denote examples in different languages. Solid and dotted arrows denote positive and negative pairs, respectively. (a) vs. (b): Multi-lingual positives by translation exhibit transitive similarity, while monolingual positives do not. (c) vs. (d): Pairwise semantic similarity scores of sampled sentences for mono- and multi-lingual, highlighting the similarity transitive similarity. Example sentences are sourced from XNLI dataset, refer to \ref{['appendix1']} for details.
  • Figure 2: Illustration of MPCL. Left: we reorganize multi-lingual data with a translation dataset to construct a training dataset with multiple positives. Sentences in the same font are translations from different languages. Right: we perform contrastive loss with multiple positive instances to update the model.
  • Figure 3: Detailed performance changes of LaBSE and mSimCSE$_{all}$ on different languages after training on multiple positive or single positive. On the horizontal axis, 0 represents the original scores without additional training. The vertical axis shows changes in performance after further training. Bars above the horizontal axis indicate improvements, while those below indicate decreases. The bold lines split the results into three different parts: STS17, STS22 and MTOP Domain Classification from left to right. Orange color highlights unseen languages.
  • Figure 4: Results of the STS17 and STS22 tasks trained on fully none-overlapping languages. We report the average scores of all language pairs included in STS17 and STS22. Error bars refer to standard deviation.
  • Figure 5: Cross-lingual similarity scores calculated by XLM-R + Multiple or XLM-R + Single for two randomly chosen examples. The example sentences are shown in the sub-figure caption. Non-English-centric language pairs are highlighted with red color.