Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do Not Change Me: On Transferring Entities Without Modification in Neural Machine Translation -- a Multilingual Perspective

Dawid Wisniewski, Mikolaj Pokrywka, Zofia Rostek

TL;DR

The paper tackles the challenge of preserving no-translate entities in neural machine translation across English, German, Polish, and Ukrainian by introducing a 36,000-sentence multilingual synthetic dataset spanning 9 entity categories. It evaluates eight popular models, including OPUS, MADLAD, MBART, M2M100, SeamlessM4T, NLLB, EuroLLM, and Google Translate, using regex-based entity detection and Levenshtein distance to analyze transfer quality and error types. Key findings show that emoji transfer is particularly problematic for several models, while long numeric sequences like IBANs or ISBNs pose higher risks of partial or altered translations; larger models (notably EuroLLM 9B) generally outperform smaller ones and approach Google Translate in performance. The study also demonstrates that focused prompts improve EuroLLM’s entity-transfer performance and that context length does not reliably improve accuracy, offering practical guidance for building robust MT systems. Overall, the work provides a new benchmark and empirical insights into how modern NMT systems handle sensitive no-translate entities, with implications for deployment in multilingual settings and for future model design.

Abstract

Current machine translation models provide us with high-quality outputs in most scenarios. However, they still face some specific problems, such as detecting which entities should not be changed during translation. In this paper, we explore the abilities of popular NMT models, including models from the OPUS project, Google Translate, MADLAD, and EuroLLM, to preserve entities such as URL addresses, IBAN numbers, or emails when producing translations between four languages: English, German, Polish, and Ukrainian. We investigate the quality of popular NMT models in terms of accuracy, discuss errors made by the models, and examine the reasons for errors. Our analysis highlights specific categories, such as emojis, that pose significant challenges for many models considered. In addition to the analysis, we propose a new multilingual synthetic dataset of 36,000 sentences that can help assess the quality of entity transfer across nine categories and four aforementioned languages.

Do Not Change Me: On Transferring Entities Without Modification in Neural Machine Translation -- a Multilingual Perspective

TL;DR

The paper tackles the challenge of preserving no-translate entities in neural machine translation across English, German, Polish, and Ukrainian by introducing a 36,000-sentence multilingual synthetic dataset spanning 9 entity categories. It evaluates eight popular models, including OPUS, MADLAD, MBART, M2M100, SeamlessM4T, NLLB, EuroLLM, and Google Translate, using regex-based entity detection and Levenshtein distance to analyze transfer quality and error types. Key findings show that emoji transfer is particularly problematic for several models, while long numeric sequences like IBANs or ISBNs pose higher risks of partial or altered translations; larger models (notably EuroLLM 9B) generally outperform smaller ones and approach Google Translate in performance. The study also demonstrates that focused prompts improve EuroLLM’s entity-transfer performance and that context length does not reliably improve accuracy, offering practical guidance for building robust MT systems. Overall, the work provides a new benchmark and empirical insights into how modern NMT systems handle sensitive no-translate entities, with implications for deployment in multilingual settings and for future model design.

Abstract

Current machine translation models provide us with high-quality outputs in most scenarios. However, they still face some specific problems, such as detecting which entities should not be changed during translation. In this paper, we explore the abilities of popular NMT models, including models from the OPUS project, Google Translate, MADLAD, and EuroLLM, to preserve entities such as URL addresses, IBAN numbers, or emails when producing translations between four languages: English, German, Polish, and Ukrainian. We investigate the quality of popular NMT models in terms of accuracy, discuss errors made by the models, and examine the reasons for errors. Our analysis highlights specific categories, such as emojis, that pose significant challenges for many models considered. In addition to the analysis, we propose a new multilingual synthetic dataset of 36,000 sentences that can help assess the quality of entity transfer across nine categories and four aforementioned languages.
Paper Structure (27 sections, 8 figures, 10 tables)

This paper contains 27 sections, 8 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Translation errors
  4. Recent neural translation models
  5. Dataset
  6. Sentences generation
  7. Representative sample selection
  8. Explorative data analysis
  9. Methodology
  10. Models selection
  11. Translation procedure
  12. Source and target sentence entity comparison
  13. Results
  14. Per-category accuracy
  15. Per-direction accuracy
  16. ...and 12 more sections

Figures (8)

  • Figure 1: Number of NLTK bird2009natural tokens in sentences expressed in a given language.
  • Figure 2: Distribution of the number of characters for each entity category.
  • Figure 3: Percentage of correctly transferred entities among different sentence lengths.
  • Figure 4: Distribution of the size of errors among models
  • Figure 5: Percentage of entities not matched in target sentences among different lengths.
  • ...and 3 more figures