GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek

TL;DR

GreekMMLU introduces the first large-scale, native-sourced multitask benchmark for Greek, enabling linguistically and culturally grounded evaluation across 45 subjects and multiple educational levels. Using a standardized evaluation framework, it reveals substantial performance gaps between frontier closed-source models and open-weight equivalents, and between Greek-adapted versus general multilingual approaches. The study demonstrates that instruction tuning and region-specific training significantly improve performance on Greek-specific domains, with calibration analyses showing that Greek-specialist models provide more reliable confidence estimates. By releasing a public dataset coupled with a private leaderboard, GreekMMLU provides a robust, contamination-resistant benchmark to drive development of truly native Greek language understanding and culturally aware AI systems.

Abstract

Large Language Models (LLMs) are commonly trained on multilingual corpora that include Greek, yet reliable evaluation benchmarks for Greek-particularly those based on authentic, native-sourced content-remain limited. Existing datasets are often machine-translated from English, failing to capture Greek linguistic and cultural characteristics. We introduce GreekMMLU, a native-sourced benchmark for massive multitask language understanding in Greek, comprising 21,805 multiple-choice questions across 45 subject areas, organized under a newly defined subject taxonomy and annotated with educational difficulty levels spanning primary to professional examinations. All questions are sourced or authored in Greek from academic, professional, and governmental exams. We publicly release 16,857 samples and reserve 4,948 samples for a private leaderboard to enable robust and contamination-resistant evaluation. Evaluations of over 80 open- and closed-source LLMs reveal substantial performance gaps between frontier and open-weight models, as well as between Greek-adapted models and general multilingual ones. Finally, we provide a systematic analysis of factors influencing performance-including model scale, adaptation, and prompting-and derive insights for improving LLM capabilities in Greek.

Figures (16)

• Figure 1: GreekMMLU task overview.
• Figure 2: Example of a mythology question from GreekMMLU. Left shows the content structure derived from native sources and right is the English translation. The bold options represent the correct answer keys.
• Figure 3: Prompt templates in Greek and English.
• Figure 4: Models' 0-shot performance on different task levels.
• Figure 5: Scaling behavior of average accuracy with respect to model size (in billions of parameters) under zero-shot and five-shot prompting.