Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multiple References with Meaningful Variations Improve Literary Machine Translation

Si Wu, John Wieting, David A. Smith

TL;DR

The paper tackles how to best exploit multiple references in literary translation by measuring semantic similarity among English paraphrases of source text translations. It introduces a par0adigm based on Paragram-SP similarity to categorize paraphrases into Low, Medium, and High, and constructs datasets with varying numbers of sources and references. Through fine-tuning mT5-large, LLaMA-2-7B, and Opus-MT (and comparing against NLLB and Google Translate) across BLEU, COMET$_{22}^{ ext{DA}}$, and chrF++, the study finds that using medium to high similarity paraphrases yields the strongest improvements, especially when training data are filtered by similarity, and that, at constant instance counts, a single-reference setup can be marginally better than multiple references. The results highlight practical guidance for literary MT: curate paraphrase inputs by semantic similarity to maximize transfer quality and consider model-specific sensitivity to fine-tuning; the approach presents a scalable way to leverage existing expert translations for improved translation quality in high-resource literary settings.

Abstract

While a source sentence can be translated in many ways, most machine translation (MT) models are trained with only a single reference. Previous work has shown that using synthetic paraphrases can improve MT. This paper investigates best practices for employing multiple references by analyzing the semantic similarity among different English translations of world literature in the Par3 dataset. We classify the semantic similarity between paraphrases into three levels: low, medium, and high, and fine-tune three different models (mT5-large, LLaMA-2-7B, and Opus-MT) for literary MT tasks. Across different models, holding the total training instances constant, single-reference but more source texts only marginally outperforms multiple-reference with half of the source texts. Moreover, when fine-tuning an LLM, using paraphrases with medium and high semantic similarity outperforms an unfiltered dataset, with improvements in BLEU (0.3-0.5), COMET (0.1-0.9), and chrF++ (0.17-0.32). Our code is publicly available on GitHub.

Multiple References with Meaningful Variations Improve Literary Machine Translation

TL;DR

The paper tackles how to best exploit multiple references in literary translation by measuring semantic similarity among English paraphrases of source text translations. It introduces a par0adigm based on Paragram-SP similarity to categorize paraphrases into Low, Medium, and High, and constructs datasets with varying numbers of sources and references. Through fine-tuning mT5-large, LLaMA-2-7B, and Opus-MT (and comparing against NLLB and Google Translate) across BLEU, COMET, and chrF++, the study finds that using medium to high similarity paraphrases yields the strongest improvements, especially when training data are filtered by similarity, and that, at constant instance counts, a single-reference setup can be marginally better than multiple references. The results highlight practical guidance for literary MT: curate paraphrase inputs by semantic similarity to maximize transfer quality and consider model-specific sensitivity to fine-tuning; the approach presents a scalable way to leverage existing expert translations for improved translation quality in high-resource literary settings.

Abstract

While a source sentence can be translated in many ways, most machine translation (MT) models are trained with only a single reference. Previous work has shown that using synthetic paraphrases can improve MT. This paper investigates best practices for employing multiple references by analyzing the semantic similarity among different English translations of world literature in the Par3 dataset. We classify the semantic similarity between paraphrases into three levels: low, medium, and high, and fine-tune three different models (mT5-large, LLaMA-2-7B, and Opus-MT) for literary MT tasks. Across different models, holding the total training instances constant, single-reference but more source texts only marginally outperforms multiple-reference with half of the source texts. Moreover, when fine-tuning an LLM, using paraphrases with medium and high semantic similarity outperforms an unfiltered dataset, with improvements in BLEU (0.3-0.5), COMET (0.1-0.9), and chrF++ (0.17-0.32). Our code is publicly available on GitHub.

Paper Structure

This paper contains 29 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Measuring semantic similarity
  4. Datasets
  5. Experiments
  6. Models
  7. mT5-large
  8. LLaMA-2-7B
  9. Opus-MT
  10. No Language Left Behind (NLLB):
  11. Google Translate:
  12. Evaluation metrics
  13. BLEU
  14. $\text{COMET}_{22}^{\text{DA}}$
  15. chrF++
  16. ...and 14 more sections

Figures (3)

  • Figure 1: An example of meaningful variations between different English translations of a Classical Chinese source text. Highlighted texts are segments where the variations are meaningful. The example source text and its English translations are from the Par3 dataset thai-etal-2022-exploring.
  • Figure 2: The distribution of sim_ p scores for the source texts in the Par3 corpus. Paraphrases with higher scores exhibit greater semantic and syntactic similarity. Upon manual inspection, we observe that the paraphrases within the red and blue vertical lines have meaningful variations: different enough to be interesting, yet similar enough to retain their nature as paraphrases.
  • Figure 3: Performance difference between different languages under different MT metrics when fine-tuning with mT5-large (1st row), LLaMA-2-7B (2nd row), and Opus-MT (3rd row). Darker dots are from Medium+High (#source$=68k$) dataset, and lighter dots are from Unfiltered dataset. The bar chart shows the number of training instances in Unfiltered dataset for each language.