Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish

Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, Hinrich Schütze

TL;DR

This work introduces the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU, to evaluate LLMs' understanding of the Turkish language and provides an in-depth analysis of the Turkish capabilities and limitations of current LLMs to provide insights for future LLMs for the Turkish language.

Abstract

Multiple choice question answering tasks evaluate the reasoning, comprehension, and mathematical abilities of Large Language Models (LLMs). While existing benchmarks employ automatic translation for multilingual evaluation, this approach is error-prone and potentially introduces culturally biased questions, especially in social sciences. We introduce the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU, to evaluate LLMs' understanding of the Turkish language. TurkishMMLU includes over 10,000 questions, covering 9 different subjects from Turkish high-school education curricula. These questions are written by curriculum experts, suitable for the high-school curricula in Turkey, covering subjects ranging from natural sciences and math questions to more culturally representative topics such as Turkish Literature and the history of the Turkish Republic. We evaluate over 20 LLMs, including multilingual open-source (e.g., Gemma, Llama, MT5), closed-source (GPT 4o, Claude, Gemini), and Turkish-adapted (e.g., Trendyol) models. We provide an extensive evaluation, including zero-shot and few-shot evaluation of LLMs, chain-of-thought reasoning, and question difficulty analysis along with model performance. We provide an in-depth analysis of the Turkish capabilities and limitations of current LLMs to provide insights for future LLMs for the Turkish language. We publicly release our code for the dataset and evaluation: https://github.com/ArdaYueksel/TurkishMMLU.

TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish

TL;DR

This work introduces the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU, to evaluate LLMs' understanding of the Turkish language and provides an in-depth analysis of the Turkish capabilities and limitations of current LLMs to provide insights for future LLMs for the Turkish language.

Abstract

Multiple choice question answering tasks evaluate the reasoning, comprehension, and mathematical abilities of Large Language Models (LLMs). While existing benchmarks employ automatic translation for multilingual evaluation, this approach is error-prone and potentially introduces culturally biased questions, especially in social sciences. We introduce the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU, to evaluate LLMs' understanding of the Turkish language. TurkishMMLU includes over 10,000 questions, covering 9 different subjects from Turkish high-school education curricula. These questions are written by curriculum experts, suitable for the high-school curricula in Turkey, covering subjects ranging from natural sciences and math questions to more culturally representative topics such as Turkish Literature and the history of the Turkish Republic. We evaluate over 20 LLMs, including multilingual open-source (e.g., Gemma, Llama, MT5), closed-source (GPT 4o, Claude, Gemini), and Turkish-adapted (e.g., Trendyol) models. We provide an extensive evaluation, including zero-shot and few-shot evaluation of LLMs, chain-of-thought reasoning, and question difficulty analysis along with model performance. We provide an in-depth analysis of the Turkish capabilities and limitations of current LLMs to provide insights for future LLMs for the Turkish language. We publicly release our code for the dataset and evaluation: https://github.com/ArdaYueksel/TurkishMMLU.
Paper Structure (15 sections, 5 figures, 7 tables)

This paper contains 15 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset
  4. Evaluation Results
  5. Experimental Setup
  6. Language Models:
  7. Few-shot Evaluation
  8. Zero-Shot Evaluation
  9. Chain-of-Thought Evaluation
  10. Difficulty Analysis
  11. Small Set - All Set Correlation
  12. Conclusion
  13. Limitations
  14. Question Examples
  15. Leaderboard

Figures (5)

  • Figure 1: The chart displays the subject distribution of TurkishMMLU. An example from our dataset shows recent multilingual LLMs struggling with a question about Turkish history.
  • Figure 2: Sample biology test from the EBA Platform (translated to English, see Figure \ref{['fig:eba_original']} in Appendix for the original Turkish test). Black boxes indicate the correctness ratio (difficulty level). Green borders appear when the user's choice matches the ground-truth answer, while red borders indicate incorrect choices.
  • Figure 3: Distribution of correctness ratios. Questions are categorized as Easy (green, top 30%), Medium (blue, middle 40%), or Hard (red, bottom 30%) based on the 30th and 70th percentiles.
  • Figure 4: 5-shot accuracy comparison of 32 open-source models for TurkishMMLUsub and TurkishMMLU (each point corresponds to an LLM). Pearson's r correlation between them is 0.999.
  • Figure 5: Sample biology test from the EBA Platform, the original version of Figure \ref{['fig:eba']} in Turkish.