Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-lingual Transfer or Machine Translation? On Data Augmentation for Monolingual Semantic Textual Similarity

Sho Hoshino, Akihiko Kato, Soichiro Murakami, Peinan Zhang

TL;DR

The paper investigates data augmentation strategies for monolingual semantic textual similarity in Japanese and Korean by comparing cross-lingual transfer from English resources against machine translation of English data. It analyzes two data domains—NLI and Wikipedia—and finds that Wikipedia-domain data, especially when used in a cross-lingual transfer setup, yields the strongest performance and can even surpass the state-of-the-art LaBSE in many cases. The results indicate that cross-lingual transfer with Wikipedia data can rival or exceed MT-based approaches, with native Wikipedia data offering additional gains, suggesting a scalable path for improving non-English sentence embeddings. The work highlights the practical value of Wikipedia as unlabeled data for training robust monolingual STS models and motivates further exploration across more languages and data-domain configurations.

Abstract

Learning better sentence embeddings leads to improved performance for natural language understanding tasks including semantic textual similarity (STS) and natural language inference (NLI). As prior studies leverage large-scale labeled NLI datasets for fine-tuning masked language models to yield sentence embeddings, task performance for languages other than English is often left behind. In this study, we directly compared two data augmentation techniques as potential solutions for monolingual STS: (a) cross-lingual transfer that exploits English resources alone as training data to yield non-English sentence embeddings as zero-shot inference, and (b) machine translation that coverts English data into pseudo non-English training data in advance. In our experiments on monolingual STS in Japanese and Korean, we find that the two data techniques yield performance on par. Rather, we find a superiority of the Wikipedia domain over the NLI domain for these languages, in contrast to prior studies that focused on NLI as training data. Combining our findings, we demonstrate that the cross-lingual transfer of Wikipedia data exhibits improved performance, and that native Wikipedia data can further improve performance for monolingual STS.

Cross-lingual Transfer or Machine Translation? On Data Augmentation for Monolingual Semantic Textual Similarity

TL;DR

The paper investigates data augmentation strategies for monolingual semantic textual similarity in Japanese and Korean by comparing cross-lingual transfer from English resources against machine translation of English data. It analyzes two data domains—NLI and Wikipedia—and finds that Wikipedia-domain data, especially when used in a cross-lingual transfer setup, yields the strongest performance and can even surpass the state-of-the-art LaBSE in many cases. The results indicate that cross-lingual transfer with Wikipedia data can rival or exceed MT-based approaches, with native Wikipedia data offering additional gains, suggesting a scalable path for improving non-English sentence embeddings. The work highlights the practical value of Wikipedia as unlabeled data for training robust monolingual STS models and motivates further exploration across more languages and data-domain configurations.

Abstract

Learning better sentence embeddings leads to improved performance for natural language understanding tasks including semantic textual similarity (STS) and natural language inference (NLI). As prior studies leverage large-scale labeled NLI datasets for fine-tuning masked language models to yield sentence embeddings, task performance for languages other than English is often left behind. In this study, we directly compared two data augmentation techniques as potential solutions for monolingual STS: (a) cross-lingual transfer that exploits English resources alone as training data to yield non-English sentence embeddings as zero-shot inference, and (b) machine translation that coverts English data into pseudo non-English training data in advance. In our experiments on monolingual STS in Japanese and Korean, we find that the two data techniques yield performance on par. Rather, we find a superiority of the Wikipedia domain over the NLI domain for these languages, in contrast to prior studies that focused on NLI as training data. Combining our findings, we demonstrate that the cross-lingual transfer of Wikipedia data exhibits improved performance, and that native Wikipedia data can further improve performance for monolingual STS.
Paper Structure (27 sections, 2 figures, 8 tables)

This paper contains 27 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Data Augmentation Techniques
  3. Experiments
  4. Setup
  5. Datasets
  6. Training
  7. Evaluation
  8. Evaluation Protocol
  9. Results
  10. Discussion
  11. Between cross-lingual transfer or machine translation, which is better?
  12. Do these approaches yield performance on par with that of a state-of-the-art multilingual model?
  13. Related Work
  14. Learning Sentence Embeddings
  15. Cross-lingual Transferability of Multilingual Models
  16. ...and 12 more sections

Figures (2)

  • Figure 1: Illustration of the two different data augmentation techniques applied from English to non-English.
  • Figure 2: 2D visualization of sentence embeddings on KLUE-STS. Different scales are used to better illustrate the isotropic nature of SimCSE fine-tuning.