Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toxicity in Multilingual Machine Translation at Scale

Marta R. Costa-jussà, Eric Smith, Christophe Ropers, Daniel Licht, Jean Maillard, Javier Ferrando, Carlos Escolano

TL;DR

The paper tackles added toxicity as a critical error in multilingual MT at scale by combining a toxicity-detection tool, the HolisticBias dataset, and the alti+ interpretability method to analyze 164 language directions. It demonstrates that added toxicity ranges from 0% to 5%, is more prevalent in low-resource languages, and is often linked to descriptor terms via the source, though hallucination also contributes. Through coarse and fine-grained analyses plus targeted human evaluation, it attributes toxicity to mistranslation and hallucination and shows that robustness and source contributions jointly modulate risk. The study offers practical mitigation strategies—data curation, hallucination reduction, and unstable-translation checks—and provides a framework for explainable toxicity in MT at scale, with code and data available on GitHub.

Abstract

Machine Translation systems can produce different types of errors, some of which are characterized as critical or catastrophic due to the specific negative impact that they can have on users. In this paper we focus on one type of critical error: added toxicity. We evaluate and analyze added toxicity when translating a large evaluation dataset (HOLISTICBIAS, over 472k sentences, covering 13 demographic axes) from English into 164 languages. An automatic toxicity evaluation shows that added toxicity across languages varies from 0% to 5%. The output languages with the most added toxicity tend to be low-resource ones, and the demographic axes with the most added toxicity include sexual orientation, gender and sex, and ability. We also perform human evaluation on a subset of 8 translation directions, confirming the prevalence of true added toxicity. We use a measurement of the amount of source contribution to the translation, where a low source contribution implies hallucination, to interpret what causes toxicity. Making use of the input attributions allows us to explain toxicity, because the source contributions significantly correlate with toxicity for 84% of languages studied. Given our findings, our recommendations to reduce added toxicity are to curate training data to avoid mistranslations, mitigate hallucination and check unstable translations.

Toxicity in Multilingual Machine Translation at Scale

TL;DR

The paper tackles added toxicity as a critical error in multilingual MT at scale by combining a toxicity-detection tool, the HolisticBias dataset, and the alti+ interpretability method to analyze 164 language directions. It demonstrates that added toxicity ranges from 0% to 5%, is more prevalent in low-resource languages, and is often linked to descriptor terms via the source, though hallucination also contributes. Through coarse and fine-grained analyses plus targeted human evaluation, it attributes toxicity to mistranslation and hallucination and shows that robustness and source contributions jointly modulate risk. The study offers practical mitigation strategies—data curation, hallucination reduction, and unstable-translation checks—and provides a framework for explainable toxicity in MT at scale, with code and data available on GitHub.

Abstract

Machine Translation systems can produce different types of errors, some of which are characterized as critical or catastrophic due to the specific negative impact that they can have on users. In this paper we focus on one type of critical error: added toxicity. We evaluate and analyze added toxicity when translating a large evaluation dataset (HOLISTICBIAS, over 472k sentences, covering 13 demographic axes) from English into 164 languages. An automatic toxicity evaluation shows that added toxicity across languages varies from 0% to 5%. The output languages with the most added toxicity tend to be low-resource ones, and the demographic axes with the most added toxicity include sexual orientation, gender and sex, and ability. We also perform human evaluation on a subset of 8 translation directions, confirming the prevalence of true added toxicity. We use a measurement of the amount of source contribution to the translation, where a low source contribution implies hallucination, to interpret what causes toxicity. Making use of the input attributions allows us to explain toxicity, because the source contributions significantly correlate with toxicity for 84% of languages studied. Given our findings, our recommendations to reduce added toxicity are to curate training data to avoid mistranslations, mitigate hallucination and check unstable translations.
Paper Structure (17 sections, 4 figures, 5 tables)

This paper contains 17 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Definitions and Background
  3. Proposed Experimental Methodology
  4. Quantification of added toxicity
  5. Phenomena causing toxicity
  6. Human evaluation of the toxicity detection methodology
  7. Conclusions
  8. Limitations
  9. Ethics statement
  10. Appendix: Model comparison
  11. Appendix: Axis Analysis
  12. Appendix: Fine-grained analysis: variation across languages
  13. Appendix: Statistical testing of source contribution level and toxicity
  14. Appendix: Robustness of translations
  15. Appendix: Language shown during human evaluations
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Examples of translations in English-to-French, English-to-Spanish or English-to-Catalan. Sentences show input attributions for bold words in the cases of hallucination (sentence 1); mistranslation (sentence 2); and a correct translation (sentence 3). We observe that the hallucination example focuses more in the target context than in the source sentence compared to the other two examples.
  • Figure 2: Levels and types of added toxicity vary greatly as a function of language and dataset.Top: The fraction of translations labeled as toxic is shown as a function of language, sorted by most to least toxic, for the Flores-200 and HolisticBias datasets. Bottom: For HolisticBias, different languages have wildly different distributions of toxic terms as a function of demographic axis, with some languages' toxicity being dominated by only one or two axes. The top 40 most frequently toxic languages are shown, in order from greatest to least toxicity.
  • Figure 3: The toxicity of descriptors in translation varies greatly as a function of both the source contribution to and the robustness of the translation.Left: the population distribution of the translations across all languages and HolisticBias sentences. Right: the rate of toxicity of translations, with white representing no samples or 0% toxicity. A high Gini impurity indicates a low robustness in the translation of descriptors across different HolisticBias nouns. Several regions have high toxicity, but many of them have few samples. However, the region bounded by the cyan box has relatively high rates of toxicity as well as high numbers of samples.
  • Figure 4: Distribution of target sentences found to contain toxic terms, split by the type of word in the source HolisticBias sentence that the toxic term is aligned to: a word in the descriptor, a word in the sentence template, or the person noun (e.g., grandma, kid). The 40 languages with the greatest prevalence of toxic sentences are shown, in order of decreasing toxicity.