Walia-LLM: Enhancing Amharic-LLaMA by Integrating Task-Specific and Generative Datasets

TL;DR

This work tackles the underrepresentation of Amharic in large-language-model fine-tuning by constructing an Amharic instruction-tuning dataset and integrating task-specific and generative data to fine-tune Llama-2-Amharic. A data-generation pipeline converts existing NLP datasets into Amharic instruction formats, while three new generation-focused datasets and translated instruction data augment the training regime. The study evaluates multiple data configurations (task-only, combined, MT) using a suite of metrics (weighted F1, Rouge, SacreBLEU, chrF++, and human assessments) and demonstrates improvements over the baseline in several classification and generation tasks, with GPT-4 often setting the performance ceiling. The results underscore the value of careful data curation and prompt design for low-resource language LLMs and provide open-source resources to extend similar efforts to other languages and datasets.

Abstract

Large language models (LLMs) have received a lot of attention in natural language processing (NLP) research because of their exceptional performance in understanding and generating human languages. However, low-resource languages are left behind due to the unavailability of resources. In this work, we focus on enhancing the LLaMA-2-Amharic model by integrating task-specific and generative datasets to improve language model performance for Amharic. We compile an Amharic instruction fine-tuning dataset and fine-tuned LLaMA-2-Amharic model. The fine-tuned model shows promising results in different NLP tasks. We open-source our dataset creation pipeline, instruction datasets, trained models, and evaluation outputs to promote language-specific studies on these models.

Figures (8)

• Figure 1: Data processing Pipeline. The pipeline creates instruction data from existing task datasets, and from generative datasets, we collected. All instructions, input, and output are in Amharic except for the MT case, as shown in the picture. The data source will not be used during training.
• Figure 2: Full training pipeline that summarizes the work done.
• Figure 3: Generation scores: weighted f1 scores for AfriSenti and MasakhaNews (left) and SacreBLEU score for Amharic QA (right)
• Figure 4: Scores for machine translation. Amharic to English translation scores (Right) and English to Amharic translation scores( left).
• Figure 5: Example data output from our dataset creation pipeline. \ref{['tab:newdata']}