Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models

David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Chukwuneke, Happy Buzaaba, Blessing Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei, Sokhar Samb, Tadesse Kebede Guge, Tombekai Vangoni Sherman, Pontus Stenetorp

TL;DR

IrokoBench introduces a human-translated, cross-task benchmark for 17 African languages to evaluate LLMs on challenging reasoning and knowledge tasks beyond basic classification. The framework spans AfriXNLI (NLI), AfriMMLU (knowledge QA), and AfriMGSM (math reasoning), and it evaluates a mix of open and proprietary LLMs under in-language and translate-test settings, including zero- and few-shot prompts. Key findings include large gaps between African languages and high-resource languages, a sizable performance gap between open and closed models, and notable gains from translate-test for English-centric models, particularly on AfriMGSM and AfriMMLU. The results underscore the need for more language-aware model development and better African-language coverage in pretraining and evaluation to ensure equitable NLP tooling for African communities.

Abstract

Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (\eg African languages) are often evaluated only on basic text classification tasks due to the lack of appropriate or comprehensive benchmarks outside of high-resource languages. In this paper, we introduce IrokoBench -- a human-translated benchmark dataset for 17 typologically-diverse low-resource African languages covering three tasks: natural language inference~(AfriXNLI), mathematical reasoning~(AfriMGSM), and multi-choice knowledge-based question answering~(AfriMMLU). We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings~(where test sets are translated into English) across 10 open and six proprietary LLMs. Our evaluation reveals a significant performance gap between high-resource languages~(such as English and French) and low-resource African languages. We observe a significant performance gap between open and proprietary models, with the highest performing open model, Gemma 2 27B only at 63\% of the best-performing proprietary model GPT-4o performance. In addition, machine translating the test set to English before evaluation helped to close the gap for larger models that are English-centric, such as Gemma 2 27B and LLaMa 3.1 70B. These findings suggest that more efforts are needed to develop and adapt LLMs for African languages.

IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models

TL;DR

IrokoBench introduces a human-translated, cross-task benchmark for 17 African languages to evaluate LLMs on challenging reasoning and knowledge tasks beyond basic classification. The framework spans AfriXNLI (NLI), AfriMMLU (knowledge QA), and AfriMGSM (math reasoning), and it evaluates a mix of open and proprietary LLMs under in-language and translate-test settings, including zero- and few-shot prompts. Key findings include large gaps between African languages and high-resource languages, a sizable performance gap between open and closed models, and notable gains from translate-test for English-centric models, particularly on AfriMGSM and AfriMMLU. The results underscore the need for more language-aware model development and better African-language coverage in pretraining and evaluation to ensure equitable NLP tooling for African communities.

Abstract

Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (\eg African languages) are often evaluated only on basic text classification tasks due to the lack of appropriate or comprehensive benchmarks outside of high-resource languages. In this paper, we introduce IrokoBench -- a human-translated benchmark dataset for 17 typologically-diverse low-resource African languages covering three tasks: natural language inference~(AfriXNLI), mathematical reasoning~(AfriMGSM), and multi-choice knowledge-based question answering~(AfriMMLU). We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings~(where test sets are translated into English) across 10 open and six proprietary LLMs. Our evaluation reveals a significant performance gap between high-resource languages~(such as English and French) and low-resource African languages. We observe a significant performance gap between open and proprietary models, with the highest performing open model, Gemma 2 27B only at 63\% of the best-performing proprietary model GPT-4o performance. In addition, machine translating the test set to English before evaluation helped to close the gap for larger models that are English-centric, such as Gemma 2 27B and LLaMa 3.1 70B. These findings suggest that more efforts are needed to develop and adapt LLMs for African languages.
Paper Structure (42 sections, 3 figures, 16 tables)

This paper contains 42 sections, 3 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multilingual Evaluation of LLMs:
  4. African Benchmark Datasets:
  5. IrokoBench
  6. Languages covered by IrokoBench
  7. Tasks covered by IrokoBench
  8. AfriXNLI
  9. AfriMMLU
  10. AfriMGSM
  11. Data collection process
  12. Translation
  13. Quality control
  14. LLMs used for evaluation
  15. Open LLMs
  16. ...and 27 more sections

Figures (3)

  • Figure 1: Task Description for IrokoBench datasets. Both AfriMGSM and AfriMMLU focus on QA, while AfriXNLI focuses on natural language inference between two pairs of sentences. For clarity, this figure provides examples in English.
  • Figure 2: Few-shot evaluation on IrokoBench, we performed 8-shots for AfriMGSM and 5-shots for the others.
  • Figure 3: Evaluation of AfriXNLI translations using AfriCOMET metric scores.