Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora

TL;DR

The paper addresses the challenge of data contamination in benchmark-based evaluation of Large Language Models within multilingual settings. It shows that translating benchmarks into Arabic can suppress traditional contamination signals even as models benefit from contaminated data, particularly for tasks with strong Arabic capabilities. To counter this blind spot, it introduces Translation-Aware Contamination Detection (TACD), a perturbation-based diagnostic that compares signals across translated variants and structural shuffles to reveal contamination patterns not detectable by English-only checks. The findings advocate for multilingual, translation-aware evaluation pipelines to improve fairness, transparency, and reproducibility in LLM assessment.

Abstract

Data contamination undermines the validity of Large Language Model evaluation by enabling models to rely on memorized benchmark content rather than true generalization. While prior work has proposed contamination detection methods, these approaches are largely limited to English benchmarks, leaving multilingual contamination poorly understood. In this work, we investigate contamination dynamics in multilingual settings by fine-tuning several open-weight LLMs on varying proportions of Arabic datasets and evaluating them on original English benchmarks. To detect memorization, we extend the Tested Slot Guessing method with a choice-reordering strategy and incorporate Min-K% probability analysis, capturing both behavioral and distributional contamination signals. Our results show that translation into Arabic suppresses conventional contamination indicators, yet models still benefit from exposure to contaminated data, particularly those with stronger Arabic capabilities. This effect is consistently reflected in rising Mink% scores and increased cross-lingual answer consistency as contamination levels grow. To address this blind spot, we propose Translation-Aware Contamination Detection, which identifies contamination by comparing signals across multiple translated benchmark variants rather than English alone. The Translation-Aware Contamination Detection reliably exposes contamination even when English-only methods fail. Together, our findings highlight the need for multilingual, translation-aware evaluation pipelines to ensure fair, transparent, and reproducible assessment of LLMs.

Figures (4)

• Figure 1: TS-Guessing across datasets. Top: MMLU (MCQ) with choice re-ordering then masking a choice; index-letter recall after shuffling is a contamination cue. Bottom: XQuAD (extractive QA) with a masked token in the question; exact recovery suggests memorization.
• Figure 2: Vocabulary size and type--token ratio (TTR) for MMLU and XQUAD.
• Figure 3: Context--question lexical overlap and answer length statistics for XQUAD.
• Figure 4: Flow diagram mapping Arabic$\!\to$English translated items (left) to MMLU subject labels (right). Band thickness encodes the number of items; higher positions on the left correspond to higher similarity bins between translated and original English embeddings (cosine space).