Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriama

Naome A. Etori, Kevin Lu, Randu Karisa, Arturs Kanepajs

TL;DR

This work addresses the lack of robust benchmarks for non-English, low-resource languages by introducing LAG-MMLU, a 500-question multilingual subset of MMLU that includes a first gold-standard Giriama dataset and a Latvian silver-standard translated dataset. It evaluates eight SOTA LLMs using the Inspect AI framework under 0-shot and 5-shot conditions with and without chain-of-thought prompting across English, Latvian, and Giriama. The results show OpenAI o1 dominating across languages, with English highest, Latvian intermediate, and Giriama notably challenging for both closed and open models, highlighting substantial gendered language-resource gaps. The work emphasizes the need for localized benchmarks and human evaluation to advance culturally contextual AI, and it provides a resource for future data collection and model adaptation in LR languages.

Abstract

As large language models (LLMs) rapidly advance, evaluating their performance is critical. LLMs are trained on multilingual data, but their reasoning abilities are mainly evaluated using English datasets. Hence, robust evaluation frameworks are needed using high-quality non-English datasets, especially low-resource languages (LRLs). This study evaluates eight state-of-the-art (SOTA) LLMs on Latvian and Giriama using a Massive Multitask Language Understanding (MMLU) subset curated with native speakers for linguistic and cultural relevance. Giriama is benchmarked for the first time. Our evaluation shows that OpenAI's o1 model outperforms others across all languages, scoring 92.8% in English, 88.8% in Latvian, and 70.8% in Giriama on 0-shot tasks. Mistral-large (35.6%) and Llama-70B IT (41%) have weak performance, on both Latvian and Giriama. Our results underscore the need for localized benchmarks and human evaluations in advancing cultural AI contextualization.

LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriama

TL;DR

This work addresses the lack of robust benchmarks for non-English, low-resource languages by introducing LAG-MMLU, a 500-question multilingual subset of MMLU that includes a first gold-standard Giriama dataset and a Latvian silver-standard translated dataset. It evaluates eight SOTA LLMs using the Inspect AI framework under 0-shot and 5-shot conditions with and without chain-of-thought prompting across English, Latvian, and Giriama. The results show OpenAI o1 dominating across languages, with English highest, Latvian intermediate, and Giriama notably challenging for both closed and open models, highlighting substantial gendered language-resource gaps. The work emphasizes the need for localized benchmarks and human evaluation to advance culturally contextual AI, and it provides a resource for future data collection and model adaptation in LR languages.

Abstract

As large language models (LLMs) rapidly advance, evaluating their performance is critical. LLMs are trained on multilingual data, but their reasoning abilities are mainly evaluated using English datasets. Hence, robust evaluation frameworks are needed using high-quality non-English datasets, especially low-resource languages (LRLs). This study evaluates eight state-of-the-art (SOTA) LLMs on Latvian and Giriama using a Massive Multitask Language Understanding (MMLU) subset curated with native speakers for linguistic and cultural relevance. Giriama is benchmarked for the first time. Our evaluation shows that OpenAI's o1 model outperforms others across all languages, scoring 92.8% in English, 88.8% in Latvian, and 70.8% in Giriama on 0-shot tasks. Mistral-large (35.6%) and Llama-70B IT (41%) have weak performance, on both Latvian and Giriama. Our results underscore the need for localized benchmarks and human evaluations in advancing cultural AI contextualization.

Paper Structure

This paper contains 35 sections, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Multilingual Models Across Cultures
  4. Multilingual LLMs Evaluation
  5. Human evaluation of multilingual models
  6. Methodology
  7. LAG-MMLU Dataset
  8. Dataset Selection Process
  9. Languages covered
  10. Latvian (lav):
  11. Giriama (nyf):
  12. Dataset collection
  13. Baseline Dataset:
  14. Machine Translations:
  15. Human-Edited Translations:
  16. ...and 20 more sections

Figures (7)

  • Figure 1: Distribution of subjects and categories of the subjects in LAG-MMLU. We included subjects with a frequency of 7 and above. 55 out of 57 subjects are represented. See more details in Table \ref{['tab:subject_distribution']} in the appendix.
  • Figure 2: Frontier LLMs in Latvian and Giriama Dataset and Benchmarking pipeline
  • Figure 3: Examples of English-Latvian translated; topics include high-school macroeconomics, high-school geography, and miscellaneous
  • Figure 4: Examples of English-Giriama translated; topics such as high-school macroeconomics, high-school geography, and miscellaneous
  • Figure 5: Performance of All Models on LAG-MMLU on English, Latvian, and Giriama (0-shot). OpenAI-o1 shows the best performance then claude 3.5 Sonnet. Mistral-large underperforms in LRLs.
  • ...and 2 more figures