Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PersianMind: A Cross-Lingual Persian-English Large Language Model

Pedram Rostami, Ali Salemi, Mohammad Javad Dousti

TL;DR

PersianMind addresses the deficiency of open-source Persian–English LLMs by building a bilingual model on LLaMa2-7B-chat, enhanced with a 10k Persian subword tokenizer and expanded embeddings. It uses LoRA-based parameter-efficient fine-tuning combined with three training phases to minimize catastrophic forgetting of English while acquiring robust Persian capabilities. The model achieves competitive results on Persian reading comprehension and multiple-choice QA, performs competitively in translation with limited parallel data, and delivers strong cross-lingual sentence embeddings, surpassing several baseline models. This work demonstrates a practical, cost-aware approach to bilingual LLM development with meaningful cross-lingual benefits and a measurable carbon footprint.

Abstract

Large language models demonstrate remarkable proficiency in various linguistic tasks and have extensive knowledge across various domains. Although they perform best in English, their ability in other languages is notable too. In contrast, open-source models, such as LLaMa, are primarily trained on English datasets, resulting in poor performance in non-English languages. In this paper, we introduce PersianMind, an open-source bilingual large language model which demonstrates comparable performance to closed-source GPT-3.5-turbo in the Persian language. By expanding LLaMa2's vocabulary with 10,000 Persian tokens and training it on a dataset comprising nearly 2 billion Persian tokens, we show that our approach preserves the model's English knowledge and employs transfer learning to excel at transferring task knowledge from one language to another.

PersianMind: A Cross-Lingual Persian-English Large Language Model

TL;DR

PersianMind addresses the deficiency of open-source Persian–English LLMs by building a bilingual model on LLaMa2-7B-chat, enhanced with a 10k Persian subword tokenizer and expanded embeddings. It uses LoRA-based parameter-efficient fine-tuning combined with three training phases to minimize catastrophic forgetting of English while acquiring robust Persian capabilities. The model achieves competitive results on Persian reading comprehension and multiple-choice QA, performs competitively in translation with limited parallel data, and delivers strong cross-lingual sentence embeddings, surpassing several baseline models. This work demonstrates a practical, cost-aware approach to bilingual LLM development with meaningful cross-lingual benefits and a measurable carbon footprint.

Abstract

Large language models demonstrate remarkable proficiency in various linguistic tasks and have extensive knowledge across various domains. Although they perform best in English, their ability in other languages is notable too. In contrast, open-source models, such as LLaMa, are primarily trained on English datasets, resulting in poor performance in non-English languages. In this paper, we introduce PersianMind, an open-source bilingual large language model which demonstrates comparable performance to closed-source GPT-3.5-turbo in the Persian language. By expanding LLaMa2's vocabulary with 10,000 Persian tokens and training it on a dataset comprising nearly 2 billion Persian tokens, we show that our approach preserves the model's English knowledge and employs transfer learning to excel at transferring task knowledge from one language to another.
Paper Structure (14 sections, 1 figure, 7 tables)

This paper contains 14 sections, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Open LLMs
  4. Parameter Efficient Fine-Tuning
  5. PersianMind Model
  6. Bilingual Tokenizer and Expansion of Embeddings
  7. Training Details
  8. Evaluations
  9. Multiple-Choice QA and Reading Comprehension
  10. Translation
  11. Semantic Textual Similarity
  12. Carbon Footprint
  13. Conclusion
  14. Generation samples from PersianMind

Figures (1)

  • Figure 1: Our Approach for Training PersianMind: We expand LLaMa2's tokenizer and embeddings with 10,000 Persian subwords. Next, we employ the LoRA technique to reduce memory usage during training, with trainable components including input and output embeddings, as well as LoRA weights. The model is trained on Persian plain text and instructions iteratively, evaluating at each step. The training objective is focused on causal language modeling, wherein the model predicts the next token based on previously observed ones.