When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation

TL;DR

This work investigates data contamination in multilingual machine translation by contrasting a FLORES-trained Bloomz model with an uncontaminated Llama control, revealing cross-direction contamination where memorized target language text boosts unseen translation directions. Through back-translation and paraphrase perturbations, as well as targeted named-entity replacements, the authors show that recall often persists despite source alterations and that target-side memorization is the primary driver of inflated MT scores. They further demonstrate cross-direction contamination by fine-tuning Llama on FLORES-eng<->xxx data and observing artificial gains in unseen directions, underscoring that contamination can propagate across translation directions. The study provides practical probes (entity replacement) and recommends validating contamination across translation directions when evaluating on multiway parallel benchmarks to ensure genuine generalization. Overall, the findings highlight the fragility of multilingual MT benchmarks to memorization and the need for careful benchmarking practices and model controls in cross-lingual evaluation.

Abstract

Large language models (LLMs) can be benchmark-contaminated, resulting in inflated scores that mask memorization as generalization, and in multilingual settings, this memorization can even transfer to "uncontaminated" languages. Using the FLORES-200 translation benchmark as a diagnostic, we study two 7-8B instruction-tuned multilingual LLMs: Bloomz, which was trained on FLORES, and Llama as an uncontaminated control. We confirm Bloomz's FLORES contamination and demonstrate that machine translation contamination can be cross-directional, artificially boosting performance in unseen translation directions due to target-side memorization. Further analysis shows that recall of memorized references often persists despite various source-side perturbation efforts like paraphrasing and named entity replacement. However, replacing named entities leads to a consistent decrease in BLEU, suggesting an effective probing method for memorization in contaminated models.

Figures (5)

• Figure 1: BLEU for Bloomz (upper left), Llama (lower left) and COMET for Bloomz (upper right), Llama (lower right). All plots have the same sources, targets, and color bar with a blue-red gradient (low-to-high).
• Figure 2: Bloomz's BLEU for {xxxpor, xxxbam, xxxasm}$\to$tam. Llama back-translated {por, bam, asm} into xxx{por, bam, asm}.
• Figure 3: BLEU scores for Llama (left) and Bloomz (right) across language pairs under different entity replacement settings (Base, One Entity, All Entities).
• Figure 4: The BLEU (left) and COMET (right) difference between the finetuned and base Llama model. Positive/negative indicates increases/decreases in the fine-tuned model compared to the base model.
• Figure 5: BLEU scores for the fine-tuned Llama (upper left), base Llama (lower left) and COMET scores for the fine-tuned Llama (upper right) and base Llama (lower right).