Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Study on the Robustness of Massively Multilingual Neural Machine Translation

Supryadi, Leiyu Pan, Deyi Xiong

TL;DR

This study tackles the robustness of massively multilingual neural machine translation for Indonesian-Chinese under naturally occurring noise. It builds a robustness evaluation benchmark of 1,001 noisy sentence pairs translated automatically by four NLLB-200 models, complemented by manual translations and MQM-based human judgments. Automatic metrics (BLEU, CHRF++) and human evaluation are analyzed to reveal correlations and model-size effects on error distributions across 10 noise types. The findings show larger MMNMT models generally improve translation quality and align better with human judgments, though noise-type patterns and trade-offs persist; the dataset and protocol provide a publicly available resource for future robustness research in MMNMT.

Abstract

Massively multilingual neural machine translation (MMNMT) has been proven to enhance the translation quality of low-resource languages. In this paper, we empirically investigate the translation robustness of Indonesian-Chinese translation in the face of various naturally occurring noise. To assess this, we create a robustness evaluation benchmark dataset for Indonesian-Chinese translation. This dataset is automatically translated into Chinese using four NLLB-200 models of different sizes. We conduct both automatic and human evaluations. Our in-depth analysis reveal the correlations between translation error types and the types of noise present, how these correlations change across different model sizes, and the relationships between automatic evaluation indicators and human evaluation indicators. The dataset is publicly available at https://github.com/tjunlp-lab/ID-ZH-MTRobustEval.

An Empirical Study on the Robustness of Massively Multilingual Neural Machine Translation

TL;DR

This study tackles the robustness of massively multilingual neural machine translation for Indonesian-Chinese under naturally occurring noise. It builds a robustness evaluation benchmark of 1,001 noisy sentence pairs translated automatically by four NLLB-200 models, complemented by manual translations and MQM-based human judgments. Automatic metrics (BLEU, CHRF++) and human evaluation are analyzed to reveal correlations and model-size effects on error distributions across 10 noise types. The findings show larger MMNMT models generally improve translation quality and align better with human judgments, though noise-type patterns and trade-offs persist; the dataset and protocol provide a publicly available resource for future robustness research in MMNMT.

Abstract

Massively multilingual neural machine translation (MMNMT) has been proven to enhance the translation quality of low-resource languages. In this paper, we empirically investigate the translation robustness of Indonesian-Chinese translation in the face of various naturally occurring noise. To assess this, we create a robustness evaluation benchmark dataset for Indonesian-Chinese translation. This dataset is automatically translated into Chinese using four NLLB-200 models of different sizes. We conduct both automatic and human evaluations. Our in-depth analysis reveal the correlations between translation error types and the types of noise present, how these correlations change across different model sizes, and the relationships between automatic evaluation indicators and human evaluation indicators. The dataset is publicly available at https://github.com/tjunlp-lab/ID-ZH-MTRobustEval.
Paper Structure (26 sections, 2 equations, 6 figures, 6 tables)

This paper contains 26 sections, 2 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. NMT Robustness and Evaluation
  4. Multilingual NMT
  5. Multilingual NMT Robustness
  6. Robustness and Evaluation Protocol
  7. Data Collection and Annotation
  8. Data Collection
  9. Data Filtering
  10. Pre-filter
  11. OOV Filter
  12. Language Model Filter
  13. Noise Identification and Annotation
  14. Translation
  15. Evaluation
  16. ...and 11 more sections

Figures (6)

  • Figure 1: Robustness evaluation and analysis protocol.
  • Figure 2: Statistics of different noise types in the curated dataset.
  • Figure 3: The change of translation error types with the increment of model parameters.
  • Figure 4: The heatmap of the occurences each of translation errors types according to the noise types.
  • Figure 5: The changing trend of the number of translation errors along with the change of model parameters on short sentences. Dot represents the downward of the trends and square represents the upward of the trends.
  • ...and 1 more figures