Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Native vs Non-Native Language Prompting: A Comparative Analysis

Mohamed Bayan Kmainasi, Rakif Khan, Ali Ezzat Shahroor, Boushra Bendou, Maram Hasanain, Firoj Alam

TL;DR

This work systematically compares native Arabic prompts, non-native (English) prompts, and mixed prompts across 11 Arabic NLP tasks using 12 datasets and three LLMs, under zero-shot and few-shot settings. It shows that non-native prompts typically yield higher performance, with mixed prompts offering gains in few-shot regimes and GPT-4o delivering the most robust results overall. The study also introduces a newly sampled test set to enable cost-effective evaluation and reports a total of 198 experimental configurations, providing a rich resource for reproducibility. The findings offer actionable guidance for prompt design in Arabic NLP and highlight the value of cross-lingual prompting strategies in low-resource contexts.

Abstract

Large language models (LLMs) have shown remarkable abilities in different fields, including standard Natural Language Processing (NLP) tasks. To elicit knowledge from LLMs, prompts play a key role, consisting of natural language instructions. Most open and closed source LLMs are trained on available labeled and unlabeled resources--digital content such as text, images, audio, and videos. Hence, these models have better knowledge for high-resourced languages but struggle with low-resourced languages. Since prompts play a crucial role in understanding their capabilities, the language used for prompts remains an important research question. Although there has been significant research in this area, it is still limited, and less has been explored for medium to low-resourced languages. In this study, we investigate different prompting strategies (native vs. non-native) on 11 different NLP tasks associated with 12 different Arabic datasets (9.7K data points). In total, we conducted 197 experiments involving 3 LLMs, 12 datasets, and 3 prompting strategies. Our findings suggest that, on average, the non-native prompt performs the best, followed by mixed and native prompts.

Native vs Non-Native Language Prompting: A Comparative Analysis

TL;DR

This work systematically compares native Arabic prompts, non-native (English) prompts, and mixed prompts across 11 Arabic NLP tasks using 12 datasets and three LLMs, under zero-shot and few-shot settings. It shows that non-native prompts typically yield higher performance, with mixed prompts offering gains in few-shot regimes and GPT-4o delivering the most robust results overall. The study also introduces a newly sampled test set to enable cost-effective evaluation and reports a total of 198 experimental configurations, providing a rich resource for reproducibility. The findings offer actionable guidance for prompt design in Arabic NLP and highlight the value of cross-lingual prompting strategies in low-resource contexts.

Abstract

Large language models (LLMs) have shown remarkable abilities in different fields, including standard Natural Language Processing (NLP) tasks. To elicit knowledge from LLMs, prompts play a key role, consisting of natural language instructions. Most open and closed source LLMs are trained on available labeled and unlabeled resources--digital content such as text, images, audio, and videos. Hence, these models have better knowledge for high-resourced languages but struggle with low-resourced languages. Since prompts play a crucial role in understanding their capabilities, the language used for prompts remains an important research question. Although there has been significant research in this area, it is still limited, and less has been explored for medium to low-resourced languages. In this study, we investigate different prompting strategies (native vs. non-native) on 11 different NLP tasks associated with 12 different Arabic datasets (9.7K data points). In total, we conducted 197 experiments involving 3 LLMs, 12 datasets, and 3 prompting strategies. Our findings suggest that, on average, the non-native prompt performs the best, followed by mixed and native prompts.
Paper Structure (32 sections, 3 figures, 3 tables)

This paper contains 32 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt and Techniques
  4. Native vs. Non-Native Language Prompting
  5. News and Social Media Analysis
  6. Datasets
  7. Hate Speech Detection:
  8. Adult Content Detection:
  9. Spam Detection:
  10. Subjectivity Identification:
  11. Propaganda Detection:
  12. Check-worthiness Detection:
  13. Factuality Detection:
  14. Claim Detection:
  15. Harmful Content Detection:
  16. ...and 17 more sections

Figures (3)

  • Figure 1: The three prompting techniques tested in this work: native, non-native, and mixed language.
  • Figure 2: Zero-shot average results. Random: random baseline. SOTA: state-of-the-art reported in abdelali-etal-2024-larabench.
  • Figure 3: 3-shot average results. Random: random baseline. SOTA: state-of-the-art reported in abdelali-etal-2024-larabench.