Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings

Md. Arid Hasan, Prerona Tarannum, Krishno Dey, Imran Razzak, Usman Naseem

TL;DR

This work addresses the evaluation gap of large language models (LLMs) in low-resource South Asian languages by creating translated sentiment and hate-speech datasets for Bangla, Hindi, and Urdu and conducting zero-shot cross-lingual assessments of GPT-4, Llama 2, and Gemini Pro. Using English as a high-resource baseline, the study shows English consistently outperforms the low-resource languages across NLI, sentiment, and hate speech tasks, with NLI achieving the strongest overall performance and GPT-4 most often leading. The results highlight that cross-lingual transfer remains limited by training data distribution and translation quality, and that some models exhibit language- and task-specific strengths (e.g., Gemini in NLI/sentiment; Llama 2 in hate speech). The findings stress the need for better multilingual data, improved translation pipelines, and prompting strategies to extend LLM capabilities to low-resource languages in real-world NLP applications.

Abstract

Large language models (LLMs) have garnered significant interest in natural language processing (NLP), particularly their remarkable performance in various downstream tasks in resource-rich languages. Recent studies have highlighted the limitations of LLMs in low-resource languages, primarily focusing on binary classification tasks and giving minimal attention to South Asian languages. These limitations are primarily attributed to constraints such as dataset scarcity, computational costs, and research gaps specific to low-resource languages. To address this gap, we present datasets for sentiment and hate speech tasks by translating from English to Bangla, Hindi, and Urdu, facilitating research in low-resource language processing. Further, we comprehensively examine zero-shot learning using multiple LLMs in English and widely spoken South Asian languages. Our findings indicate that GPT-4 consistently outperforms Llama 2 and Gemini, with English consistently demonstrating superior performance across diverse tasks compared to low-resource languages. Furthermore, our analysis reveals that natural language inference (NLI) exhibits the highest performance among the evaluated tasks, with GPT-4 demonstrating superior capabilities.

Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings

TL;DR

This work addresses the evaluation gap of large language models (LLMs) in low-resource South Asian languages by creating translated sentiment and hate-speech datasets for Bangla, Hindi, and Urdu and conducting zero-shot cross-lingual assessments of GPT-4, Llama 2, and Gemini Pro. Using English as a high-resource baseline, the study shows English consistently outperforms the low-resource languages across NLI, sentiment, and hate speech tasks, with NLI achieving the strongest overall performance and GPT-4 most often leading. The results highlight that cross-lingual transfer remains limited by training data distribution and translation quality, and that some models exhibit language- and task-specific strengths (e.g., Gemini in NLI/sentiment; Llama 2 in hate speech). The findings stress the need for better multilingual data, improved translation pipelines, and prompting strategies to extend LLM capabilities to low-resource languages in real-world NLP applications.

Abstract

Large language models (LLMs) have garnered significant interest in natural language processing (NLP), particularly their remarkable performance in various downstream tasks in resource-rich languages. Recent studies have highlighted the limitations of LLMs in low-resource languages, primarily focusing on binary classification tasks and giving minimal attention to South Asian languages. These limitations are primarily attributed to constraints such as dataset scarcity, computational costs, and research gaps specific to low-resource languages. To address this gap, we present datasets for sentiment and hate speech tasks by translating from English to Bangla, Hindi, and Urdu, facilitating research in low-resource language processing. Further, we comprehensively examine zero-shot learning using multiple LLMs in English and widely spoken South Asian languages. Our findings indicate that GPT-4 consistently outperforms Llama 2 and Gemini, with English consistently demonstrating superior performance across diverse tasks compared to low-resource languages. Furthermore, our analysis reveals that natural language inference (NLI) exhibits the highest performance among the evaluated tasks, with GPT-4 demonstrating superior capabilities.
Paper Structure (20 sections, 2 figures, 9 tables)

This paper contains 20 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Results and Discussion
  5. Conclusion
  6. Prompts and Safety Setting
  7. Experimental Details and Results
  8. Experimental Settings
  9. Data
  10. NLI Task:
  11. Sentiment Task:
  12. Hate Speech Task:
  13. Translation:
  14. Data Pre-processing
  15. Evaluation Metrics
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Number of unpredicted samples by GPT-4 and Llama 2. Note that we only include the languages and models from the tasks with unpredicted samples.
  • Figure 2: Number of samples that are blocked by Gemini.