Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study

Hadeel Saadany, Ashraf Tantawy, Constantin Orasan

TL;DR

The paper investigates cyber risks associated with fluent yet dangerous machine-translation errors in Arabic-English mental-health tweets. It defines MT critical errors, builds an open dataset (from AraDepSu) annotated for such errors, and demonstrates how mainstream MT quality metrics can fail to penalize these mistakes in safety-critical tasks like depression and suicidal-ideation detection. Through error taxonomy and fine-grained analysis of dialectal, grammatical, orthographic, lexical, and deletion errors, the authors reveal that current metrics often provide false confidence when critical errors are present. The work argues for new evaluation measures and increased awareness of MT limitations in high-risk domains, with implications for safer deployment of MT tools in mental health monitoring and other safety-critical NLP applications.

Abstract

With the advent of Neural Machine Translation (NMT) systems, the MT output has reached unprecedented accuracy levels which resulted in the ubiquity of MT tools on almost all online platforms with multilingual content. However, NMT systems, like other state-of-the-art AI generative systems, are prone to errors that are deemed machine hallucinations. The problem with NMT hallucinations is that they are remarkably \textit{fluent} hallucinations. Since they are trained to produce grammatically correct utterances, NMT systems are capable of producing mistranslations that are too fluent to be recognised by both users of the MT tool, as well as by automatic quality metrics that are used to gauge their performance. In this paper, we introduce an authentic dataset of machine translation critical errors to point to the ethical and safety issues involved in the common use of MT. The dataset comprises mistranslations of Arabic mental health postings manually annotated with critical error types. We also show how the commonly used quality metrics do not penalise critical errors and highlight this as a critical issue that merits further attention from researchers.

Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study

TL;DR

The paper investigates cyber risks associated with fluent yet dangerous machine-translation errors in Arabic-English mental-health tweets. It defines MT critical errors, builds an open dataset (from AraDepSu) annotated for such errors, and demonstrates how mainstream MT quality metrics can fail to penalize these mistakes in safety-critical tasks like depression and suicidal-ideation detection. Through error taxonomy and fine-grained analysis of dialectal, grammatical, orthographic, lexical, and deletion errors, the authors reveal that current metrics often provide false confidence when critical errors are present. The work argues for new evaluation measures and increased awareness of MT limitations in high-risk domains, with implications for safer deployment of MT tools in mental health monitoring and other safety-critical NLP applications.

Abstract

With the advent of Neural Machine Translation (NMT) systems, the MT output has reached unprecedented accuracy levels which resulted in the ubiquity of MT tools on almost all online platforms with multilingual content. However, NMT systems, like other state-of-the-art AI generative systems, are prone to errors that are deemed machine hallucinations. The problem with NMT hallucinations is that they are remarkably \textit{fluent} hallucinations. Since they are trained to produce grammatically correct utterances, NMT systems are capable of producing mistranslations that are too fluent to be recognised by both users of the MT tool, as well as by automatic quality metrics that are used to gauge their performance. In this paper, we introduce an authentic dataset of machine translation critical errors to point to the ethical and safety issues involved in the common use of MT. The dataset comprises mistranslations of Arabic mental health postings manually annotated with critical error types. We also show how the commonly used quality metrics do not penalise critical errors and highlight this as a critical issue that merits further attention from researchers.
Paper Structure (19 sections, 2 figures, 1 table)

This paper contains 19 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. MT Critical Errors
  3. Data Compiling and Annotation
  4. Error Analysis
  5. Dialectical Language
  6. Grammar Type Errors
  7. Orthographic and Diacritic Errors
  8. Mistranslation of Vocabulary
  9. Deletion Errors
  10. Quality Metrics Performance
  11. SacreBLEU
  12. Meteor
  13. BERTScore
  14. Rouge
  15. GoogleBLEU
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Frequency of Error Types
  • Figure 2: Metric scores for different types of errors