A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism

TL;DR

The paper investigates how long-term, low-cost machine translation reshapes web content by constructing MWccMatrix, the largest known multi-way parallel corpus, from ccMatrix. It finds that a large fraction of content in lower-resource languages is MT-generated, with substantial multi-way parallelism, shorter and simpler sentences, lower translation quality, and a strong topical bias toward Conversation/Opinion content originating in English. These findings have significant implications for training multilingual large language models, suggesting that MT-generated data can affect fluency, accuracy, and topic distribution, and highlighting the need for robust MT-detection and data-filtering strategies. The work provides a scalable methodology and public resources to quantify and inspect web-scale multilingual data quality.

Abstract

We show that content on the web is often translated into many languages, and the low quality of these multi-way translations indicates they were likely created using Machine Translation (MT). Multi-way parallel, machine generated content not only dominates the translations in lower resource languages; it also constitutes a large fraction of the total web content in those languages. We also find evidence of a selection bias in the type of content which is translated into many languages, consistent with low quality English content being translated en masse into many lower resource languages, via MT. Our work raises serious concerns about training models such as multilingual large language models on both monolingual and bilingual data scraped from the web.

Figures (8)

• Figure 1: The more languages a sentence has been translated into ("Multi-way Parallelism"), the lower quality the translations are, suggesting a higher prevalence of machine translation. See \ref{['analysis:bitext']} for more details.
• Figure 2: Fraction of the total monolingual data used to create ccMatrix with one or more translation, in the 54 languages for which we can compute it. See \ref{['appendix:scatterplot']} for a larger plot with language codes.
• Figure 3: Fraction of parallel data in each language which is multi-way parallel (bar chart, right y-axis) and number of unique sentences (solid black line, left y-axis, log scale) by language (x-axis). Low-resource languages exhibit a dramatic increase in the amount of highly multi-way parallel data (hatched gray bars).
• Figure 4: Median perplexity (measured by GPT-2) vs multi-way parallelism, in English. We stratify by sentence length, as shorter content tends to have higher perplexity, likely due to GPT-2 having no or little context for predicting the first few words. More multi-way parallel data has lower perplexity (i.e. easier to predict).
• Figure 5: CometQE score vs sentence length (average of source and target, in characters), for Fr-De. Other language pairs (not shown) are very similar. Quality differences between levels of multi-way parallelism holds across sentence length.