Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Open-Source Large Language Models for Persian in Zero-Shot and Few-Shot Learning

Mahdi Cherakhloo, Arash Abbasi, Mohammad Saeid Sarafraz, Bijan Vosoughi Vahdat

TL;DR

This study benchmarks eleven open-source LLMs on Persian language tasks under zero-shot and few-shot learning, using datasets such as ParsiNLU, ArmanEmo, ArmanNER, Persian MMLU, and Persian news summarization benchmarks. Gemma2 consistently outperforms peers across most tasks and settings, highlighting strong cross-lingual generalization; however, token-level tasks like NER remain challenging due to Persian's morphology and orthography. Few-shot prompting yields substantial gains, particularly in reasoning and comprehension, while domain variability in Persian MMLU reveals uneven knowledge representation across subjects. The results underscore the value of cross-lingual transfer in open-source LLMs, while pinpointing essential gaps and guiding future work toward Persian-specific resources, advanced prompting, and broader benchmark coverage to advance equitable multilingual NLP.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across numerous languages; however, their effectiveness in low-resource languages like Persian requires thorough investigation. This paper presents a comprehensive benchmark of several open-source LLMs for Persian Natural Language Processing (NLP) tasks, utilizing both zero-shot and few-shot learning paradigms. We evaluate models across a range of tasks including sentiment analysis, named entity recognition, reading comprehension, and question answering, using established Persian datasets such as ParsiNLU and ArmanEmo. Our methodology encompasses rigorous experimental setups for both zero-shot and few-shot scenarios, employing metrics such as Accuracy, F1-score, BLEU, and ROUGE for performance evaluation. The results reveal that Gemma 2 consistently outperforms other models across nearly all tasks in both learning paradigms, with particularly strong performance in complex reasoning tasks. However, most models struggle with token-level understanding tasks like Named Entity Recognition, highlighting specific challenges in Persian language processing. This study contributes to the growing body of research on multilingual LLMs, providing valuable insights into their performance in Persian and offering a benchmark for future model development.

Benchmarking Open-Source Large Language Models for Persian in Zero-Shot and Few-Shot Learning

TL;DR

This study benchmarks eleven open-source LLMs on Persian language tasks under zero-shot and few-shot learning, using datasets such as ParsiNLU, ArmanEmo, ArmanNER, Persian MMLU, and Persian news summarization benchmarks. Gemma2 consistently outperforms peers across most tasks and settings, highlighting strong cross-lingual generalization; however, token-level tasks like NER remain challenging due to Persian's morphology and orthography. Few-shot prompting yields substantial gains, particularly in reasoning and comprehension, while domain variability in Persian MMLU reveals uneven knowledge representation across subjects. The results underscore the value of cross-lingual transfer in open-source LLMs, while pinpointing essential gaps and guiding future work toward Persian-specific resources, advanced prompting, and broader benchmark coverage to advance equitable multilingual NLP.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across numerous languages; however, their effectiveness in low-resource languages like Persian requires thorough investigation. This paper presents a comprehensive benchmark of several open-source LLMs for Persian Natural Language Processing (NLP) tasks, utilizing both zero-shot and few-shot learning paradigms. We evaluate models across a range of tasks including sentiment analysis, named entity recognition, reading comprehension, and question answering, using established Persian datasets such as ParsiNLU and ArmanEmo. Our methodology encompasses rigorous experimental setups for both zero-shot and few-shot scenarios, employing metrics such as Accuracy, F1-score, BLEU, and ROUGE for performance evaluation. The results reveal that Gemma 2 consistently outperforms other models across nearly all tasks in both learning paradigms, with particularly strong performance in complex reasoning tasks. However, most models struggle with token-level understanding tasks like Named Entity Recognition, highlighting specific challenges in Persian language processing. This study contributes to the growing body of research on multilingual LLMs, providing valuable insights into their performance in Persian and offering a benchmark for future model development.
Paper Structure (34 sections, 3 figures, 14 tables)

This paper contains 34 sections, 3 figures, 14 tables.

Table of Contents

  1. Introduction
  2. RELATED WORKS
  3. LLM Evaluation Methodologies
  4. Multilingual Capabilities of LLMs
  5. Persian NLP Research and Resources
  6. Zero-Shot and Few-Shot Learning in Non-English Languages
  7. Methodology
  8. Models Evaluated
  9. Datasets and Tasks
  10. ParsiNLU
  11. ArmanEmo
  12. ArmanNER
  13. Persian MMLU
  14. Persian News Summary and XLSummary
  15. Evaluation Metrics
  16. ...and 19 more sections

Figures (3)

  • Figure 1: This bar chart compares the average performance scores of various language models on Persian language tasks under two different conditions: few-shot learning (darker bars) and zero-shot learning (lighter bars)
  • Figure 2: This bar chart compares the average performance scores of various language models on Persian language tasks under few-shot learning
  • Figure 3: This bar chart compares the average performance scores of various language models on Persian language tasks under zero-shot learning