Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP

Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, Muhammad Abdul-Mageed

TL;DR

This work provides the first large-scale evaluation of ChatGPT on Arabic NLP, benchmarking 44 Arabic tasks across 60+ datasets to reveal systematic gaps relative to Arabic finetuned models. It compares ChatGPT and GPT-4 against Arabic-specific baselines (MARBERTV2, AraT5) and analyzes dialectal Arabic (MSA vs DA), prompt design effects, and human vs GPT-4-based evaluation. The findings show that ChatGPT generally underperforms compared with smaller, finetuned Arabic models, with dialectal Arabic presenting the largest challenges; GPT-4 offers stronger performance and better alignment with human judgments for evaluation. The study highlights a substantial need for Arabic-focused LLMs and suggests GPT-4-based evaluation as a scalable proxy, while cautioning about reliance on automated metrics for nuanced generation tasks.

Abstract

ChatGPT's emergence heralds a transformative phase in NLP, particularly demonstrated through its excellent performance on many English benchmarks. However, the model's efficacy across diverse linguistic contexts remains largely uncharted territory. This work aims to bridge this knowledge gap, with a primary focus on assessing ChatGPT's capabilities on Arabic languages and dialectal varieties. Our comprehensive study conducts a large-scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets. To our knowledge, this marks the first extensive performance analysis of ChatGPT's deployment in Arabic NLP. Our findings indicate that, despite its remarkable performance in English, ChatGPT is consistently surpassed by smaller models that have undergone finetuning on Arabic. We further undertake a meticulous comparison of ChatGPT and GPT-4's Modern Standard Arabic (MSA) and Dialectal Arabic (DA), unveiling the relative shortcomings of both models in handling Arabic dialects compared to MSA. Although we further explore and confirm the utility of employing GPT-4 as a potential alternative for human evaluation, our work adds to a growing body of research underscoring the limitations of ChatGPT.

GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP

TL;DR

This work provides the first large-scale evaluation of ChatGPT on Arabic NLP, benchmarking 44 Arabic tasks across 60+ datasets to reveal systematic gaps relative to Arabic finetuned models. It compares ChatGPT and GPT-4 against Arabic-specific baselines (MARBERTV2, AraT5) and analyzes dialectal Arabic (MSA vs DA), prompt design effects, and human vs GPT-4-based evaluation. The findings show that ChatGPT generally underperforms compared with smaller, finetuned Arabic models, with dialectal Arabic presenting the largest challenges; GPT-4 offers stronger performance and better alignment with human judgments for evaluation. The study highlights a substantial need for Arabic-focused LLMs and suggests GPT-4-based evaluation as a scalable proxy, while cautioning about reliance on automated metrics for nuanced generation tasks.

Abstract

ChatGPT's emergence heralds a transformative phase in NLP, particularly demonstrated through its excellent performance on many English benchmarks. However, the model's efficacy across diverse linguistic contexts remains largely uncharted territory. This work aims to bridge this knowledge gap, with a primary focus on assessing ChatGPT's capabilities on Arabic languages and dialectal varieties. Our comprehensive study conducts a large-scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets. To our knowledge, this marks the first extensive performance analysis of ChatGPT's deployment in Arabic NLP. Our findings indicate that, despite its remarkable performance in English, ChatGPT is consistently surpassed by smaller models that have undergone finetuning on Arabic. We further undertake a meticulous comparison of ChatGPT and GPT-4's Modern Standard Arabic (MSA) and Dialectal Arabic (DA), unveiling the relative shortcomings of both models in handling Arabic dialects compared to MSA. Although we further explore and confirm the utility of employing GPT-4 as a potential alternative for human evaluation, our work adds to a growing body of research underscoring the limitations of ChatGPT.
Paper Structure (35 sections, 8 figures, 11 tables)

This paper contains 35 sections, 8 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Performance on English.
  4. Performance on Multilingual Tasks.
  5. Datasets
  6. Prompt Design
  7. Experiments
  8. Evaluation on NLU Tasks
  9. Evaluation on NLG Tasks
  10. Performance on Dialectal Arabic
  11. Human Evaluation
  12. Using GPT-4 to Evaluate Responses
  13. Does GPT-4 eval correlate with human eval?
  14. Conclusion
  15. Limitations
  16. ...and 20 more sections

Figures (8)

  • Figure 1: Experimental setup for our evaluation. We evaluate ChatGPT on $44$ Arabic NLP tasks.
  • Figure 2: Prompt templates for different tasks.
  • Figure 3: Comparison between ChatGPT and GPT-4 on MSA vs DA in macro-F1 for $11$ ORCA tasks.
  • Figure 4: ChatGPT and GPT-4 on dialectal MT.
  • Figure 5: Evaluation of the models' responses by human and GPT-4. A is the best, and D is the worst rating. MT - Machine Translation, CST - Code Switched Translation, DG - Dialogue Generation.
  • ...and 3 more figures