Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lugha-Llama: Adapting Large Language Models for African Languages

Happy Buzaaba, Alexander Wettig, David Ifeoluwa Adelani, Christiane Fellbaum

TL;DR

Low-resource African languages remain underrepresented in large language models. The authors propose Lugha-Llama, an Africa-centric continuation of Llama-3.1-8B trained on 10B multilingual tokens, and find that integrating high-quality English educational content with African-language data yields substantial gains on IrokoBench and AfriQA. A Swahili-focused case study shows that the semantic content of training data, rather than its language of origin, drives performance improvements, suggesting large-scale machine translation as a means to close data-quality gaps. The work provides open-source Lugha-Llama models and a Swahili-translated educational corpus, while noting limitations in evaluation scope and high computational costs.

Abstract

Large language models (LLMs) have achieved impressive results in a wide range of natural language applications. However, they often struggle to recognize low-resource languages, in particular African languages, which are not well represented in large training corpora. In this paper, we consider how to adapt LLMs to low-resource African languages. We find that combining curated data from African languages with high-quality English educational texts results in a training mix that substantially improves the model's performance on these languages. On the challenging IrokoBench dataset, our models consistently achieve the best performance amongst similarly sized baselines, particularly on knowledge-intensive multiple-choice questions (AfriMMLU). Additionally, on the cross-lingual question answering benchmark AfriQA, our models outperform the base model by over 10%. To better understand the role of English data during training, we translate a subset of 200M tokens into Swahili language and perform an analysis which reveals that the content of these data is primarily responsible for the strong performance. We release our models and data to encourage future research on African languages.

Lugha-Llama: Adapting Large Language Models for African Languages

TL;DR

Low-resource African languages remain underrepresented in large language models. The authors propose Lugha-Llama, an Africa-centric continuation of Llama-3.1-8B trained on 10B multilingual tokens, and find that integrating high-quality English educational content with African-language data yields substantial gains on IrokoBench and AfriQA. A Swahili-focused case study shows that the semantic content of training data, rather than its language of origin, drives performance improvements, suggesting large-scale machine translation as a means to close data-quality gaps. The work provides open-source Lugha-Llama models and a Swahili-translated educational corpus, while noting limitations in evaluation scope and high computational costs.

Abstract

Large language models (LLMs) have achieved impressive results in a wide range of natural language applications. However, they often struggle to recognize low-resource languages, in particular African languages, which are not well represented in large training corpora. In this paper, we consider how to adapt LLMs to low-resource African languages. We find that combining curated data from African languages with high-quality English educational texts results in a training mix that substantially improves the model's performance on these languages. On the challenging IrokoBench dataset, our models consistently achieve the best performance amongst similarly sized baselines, particularly on knowledge-intensive multiple-choice questions (AfriMMLU). Additionally, on the cross-lingual question answering benchmark AfriQA, our models outperform the base model by over 10%. To better understand the role of English data during training, we translate a subset of 200M tokens into Swahili language and perform an analysis which reveals that the content of these data is primarily responsible for the strong performance. We release our models and data to encourage future research on African languages.

Paper Structure

This paper contains 23 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Low-resource languages.
  4. Multi-lingual language models.
  5. Data Quality.
  6. Experimental Setup
  7. Training.
  8. Evaluation.
  9. Baseline models.
  10. Results
  11. Lugha-Llamas achieve strong results.
  12. Addition of English data.
  13. Case Study on Swahili Data Quality
  14. Conclusion
  15. Narrow evaluation.
  16. ...and 8 more sections

Figures (1)

  • Figure 1: Tokens per language in the WURA corpus oladipo2023better and the 6B-token training data for training using UniMax sampling chung2023unimax.