Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ara-HOPE: Human-Centric Post-Editing Evaluation for Dialectal Arabic to Modern Standard Arabic Translation

Abdullah Alabdullah, Lifeng Han, Chenghua Lin

TL;DR

Ara-HOPE addresses the difficulty of evaluating DA to MSA translations by introducing a dialect-specific human post-editing framework. It defines a five-category error taxonomy and a decision-tree annotation protocol to capture fluency, meaning transfer, and adaptation errors, enabling detailed cross-system comparisons among Jais, GPT-3.5, and NLLB-200. Through a 200-tweet Levantine DA-MSA dataset and rigorous IAA analyses, the study shows dialect-specific terminology and semantic preservation as persistent challenges and demonstrates the framework's utility for diagnosing system weaknesses and guiding improvements. The approach offers a reproducible standard for dialect-aware MT evaluation with practical guidelines for token-level post-editing and error analysis.

Abstract

Dialectal Arabic to Modern Standard Arabic (DA-MSA) translation is a challenging task in Machine Translation (MT) due to significant lexical, syntactic, and semantic divergences between Arabic dialects and MSA. Existing automatic evaluation metrics and general-purpose human evaluation frameworks struggle to capture dialect-specific MT errors, hindering progress in translation assessment. This paper introduces Ara-HOPE, a human-centric post-editing evaluation framework designed to systematically address these challenges. The framework includes a five-category error taxonomy and a decision-tree annotation protocol. Through comparative evaluation of three MT systems (Arabic-centric Jais, general-purpose GPT-3.5, and baseline NLLB-200), Ara-HOPE effectively highlights systematic performance differences between these systems. The results show that dialect-specific terminology and semantic preservation remain the most persistent challenges in DA-MSA translation. Ara-HOPE establishes a new framework for evaluating Dialectal Arabic MT quality and provides actionable guidance for improving dialect-aware MT systems.

Ara-HOPE: Human-Centric Post-Editing Evaluation for Dialectal Arabic to Modern Standard Arabic Translation

TL;DR

Ara-HOPE addresses the difficulty of evaluating DA to MSA translations by introducing a dialect-specific human post-editing framework. It defines a five-category error taxonomy and a decision-tree annotation protocol to capture fluency, meaning transfer, and adaptation errors, enabling detailed cross-system comparisons among Jais, GPT-3.5, and NLLB-200. Through a 200-tweet Levantine DA-MSA dataset and rigorous IAA analyses, the study shows dialect-specific terminology and semantic preservation as persistent challenges and demonstrates the framework's utility for diagnosing system weaknesses and guiding improvements. The approach offers a reproducible standard for dialect-aware MT evaluation with practical guidelines for token-level post-editing and error analysis.

Abstract

Dialectal Arabic to Modern Standard Arabic (DA-MSA) translation is a challenging task in Machine Translation (MT) due to significant lexical, syntactic, and semantic divergences between Arabic dialects and MSA. Existing automatic evaluation metrics and general-purpose human evaluation frameworks struggle to capture dialect-specific MT errors, hindering progress in translation assessment. This paper introduces Ara-HOPE, a human-centric post-editing evaluation framework designed to systematically address these challenges. The framework includes a five-category error taxonomy and a decision-tree annotation protocol. Through comparative evaluation of three MT systems (Arabic-centric Jais, general-purpose GPT-3.5, and baseline NLLB-200), Ara-HOPE effectively highlights systematic performance differences between these systems. The results show that dialect-specific terminology and semantic preservation remain the most persistent challenges in DA-MSA translation. Ara-HOPE establishes a new framework for evaluating Dialectal Arabic MT quality and provides actionable guidance for improving dialect-aware MT systems.
Paper Structure (18 sections, 5 figures, 2 tables)

This paper contains 18 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Advancements in DA translation
  4. DA translation evaluation
  5. Methodology Design
  6. Framework Development
  7. Dataset Preparation
  8. Translation Systems Selection
  9. Pilot Testing and Framework Refinement
  10. Questionnaire Design
  11. Results Analysis
  12. Inter Annotator Agreement
  13. Quantitative Error Analysis
  14. Qualitative Error Analysis
  15. Analysis of TRM Errors
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Example layout of the annotation questionnaire used for evaluating DA-MSA translations across fluency, meaning transfer, and adaptation error categories.
  • Figure 2: Comparison of error severity distribution among Jais, GPT-3.5, and NLLB-200, highlighting proportions of major, minor, and no-error translations.
  • Figure 3: Comparison of Jais, GPT-3.5, and NLLB-200 models’ error scores across error categories
  • Figure 4: Visualization of accumulated error scores across fluency, meaning transfer, and adaptation error categories for Jais, GPT-3.5, and NLLB-200 in DA-MSA translation.
  • Figure 5: Structured decision tree guiding annotators through error classification for evaluating DA-MSA translations using the Ara-HOPE framework.