Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approach

TL;DR

This work tackles the scarcity of Persian instruction-following data by introducing FarsInstruct, a large-scale, human-annotated Persian instruction dataset built from 21 public datasets and translations from P3, featuring 197 templates across 21 datasets and two prompt classes (categorization and generation). To enable robust multi-task adaptation for LoRA-based models, the authors propose Co-CoLA, a Continual-Chain of LoRA framework that interleaves tuning, merging, and expanding with rehearsal on replayed past tasks, leveraging memory $T_i^r = T_i \cup \left( \sum_{j=1}^{i-1} rT_j \right)$ and the weight update $W' = W + BA$. Evaluated on Ava-Llama-3-based Persian LLMs against monolingual and multilingual baselines, Co-CoLA achieves superior ROUGE-L F1 scores across held-in and held-out Persian tasks, including notable gains on ParsiNLU Entailment and Query Paraphrasing, often with fewer parameters than larger models like Aya. The open-source FarsInstruct dataset and the Co-CoLA training paradigm offer a practical path toward stronger Persian instruction-following capabilities and broader multilingual NLP impact.

Abstract

Instruction-tuned large language models have demonstrated remarkable capabilities in following human instructions across various domains. However, their proficiency remains notably deficient in many low-resource languages. To address this challenge, we begin by introducing FarsInstruct a comprehensive instruction dataset designed to enhance the instruction following ability of large language models specifically for the Persian language a significant yet underrepresented language globally. FarsInstruct encompasses a wide range of task types and datasets, each containing a mix of straightforward to complex manual written instructions, as well as translations from the Public Pool of Prompts, ensuring a rich linguistic and cultural representation. Furthermore, we introduce Co-CoLA, a framework designed to enhance the multi-task adaptability of LoRA-tuned models. Through extensive experimental analyses, our study showcases the effectiveness of the FarsInstruct dataset coupled with training by the Co-CoLA framework, in improving the performance of large language models within the Persian context. As of the current writing, FarsInstruct comprises 197 templates across 21 distinct datasets, and we intend to update it consistently, thus augmenting its applicability.

Figures (7)

• Figure 1: An example of the prompts utilized in the training process. The Persian version of the prompt is employed for training purposes, while the translated English version is provided to enhance comprehension. The instruction component is highlighted in black, the data fields are marked in orange, and the target answer is indicated in gray. In Appendix \ref{['sec:appendix_prompts']}, this example is shown in the PromptSource environment.
• Figure 2: The detailed depiction of 11 task types utilized in our dataset. Each box within the figure lists the specific datasets associated with the respective task type. Datasets designated for training are highlighted in blue, and those reserved for testing are marked in orange. Additionally, manual datasets, which have been specifically curated and prompted by our team, are enclosed with solid borders. In contrast, datasets that have been translated from English to Persian are enclosed with dashed borders.
• Figure 3: Distribution of NLP tasks across the FarsInstruct dataset, highlighting the expanded data volumes after applying prompt templates and the number of prompts designed per task type. For each dataset, the final size is determined by multiplying the number of samples (N) by the number of prompt templates (M), resulting in a dataset size of N*M.
• Figure 4: The Continual-Chain of LoRA training procedure, containing Tuning, Merging, and Expanding. In Step 1, the pretrained language model is LoRA-tuned on dataset_1, with the replay memory initialized as empty and merged. In Step 2, the model is expanded with a new LoRA module and further tuned on a subset of dataset_1, determined by the rehearsal hyperparameter, alongside dataset_2, preparing it for Step 3. This process is iteratively repeated in subsequent steps.
• Figure 5: Comparative performance of different models on Persian language tasks using the ROUGE-L metric. The bar chart depicts the superior performance of Co-CoLA across multiple tasks, particularly excelling in the ParsiNLU Entailment task.