Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Estonian Native Large Language Model Benchmark

Helena Grete Lillepalu, Tanel Alumäe

TL;DR

This paper addresses the lack of Estonian LLM benchmarks by introducing a seven-task benchmark built from native Estonian sources, and evaluating 6 base and 26 instruction-tuned models. It employs both human judgments and an LLM-as-a-judge (Claude Sonnet) to assess performance, finding high agreement between evaluators and notable cross-task correlations. The results reveal substantial benefits from language-specific fine-tuning and model scaling, while validating automated evaluation as viable in a low-resource language context. The authors also release benchmarking code and datasets to advance Estonian NLP research.

Abstract

The availability of LLM benchmarks for the Estonian language is limited, and a comprehensive evaluation comparing the performance of different LLMs on Estonian tasks has yet to be conducted. We introduce a new benchmark for evaluating LLMs in Estonian, based on seven diverse datasets. These datasets assess general and domain-specific knowledge, understanding of Estonian grammar and vocabulary, summarization abilities, contextual comprehension, and more. The datasets are all generated from native Estonian sources without using machine translation. We compare the performance of base models, instruction-tuned open-source models, and commercial models. Our evaluation includes 6 base models and 26 instruction-tuned models. To assess the results, we employ both human evaluation and LLM-as-a-judge methods. Human evaluation scores showed moderate to high correlation with benchmark evaluations, depending on the dataset. Claude 3.7 Sonnet, used as an LLM judge, demonstrated strong alignment with human ratings, indicating that top-performing LLMs can effectively support the evaluation of Estonian-language models.

Estonian Native Large Language Model Benchmark

TL;DR

This paper addresses the lack of Estonian LLM benchmarks by introducing a seven-task benchmark built from native Estonian sources, and evaluating 6 base and 26 instruction-tuned models. It employs both human judgments and an LLM-as-a-judge (Claude Sonnet) to assess performance, finding high agreement between evaluators and notable cross-task correlations. The results reveal substantial benefits from language-specific fine-tuning and model scaling, while validating automated evaluation as viable in a low-resource language context. The authors also release benchmarking code and datasets to advance Estonian NLP research.

Abstract

The availability of LLM benchmarks for the Estonian language is limited, and a comprehensive evaluation comparing the performance of different LLMs on Estonian tasks has yet to be conducted. We introduce a new benchmark for evaluating LLMs in Estonian, based on seven diverse datasets. These datasets assess general and domain-specific knowledge, understanding of Estonian grammar and vocabulary, summarization abilities, contextual comprehension, and more. The datasets are all generated from native Estonian sources without using machine translation. We compare the performance of base models, instruction-tuned open-source models, and commercial models. Our evaluation includes 6 base models and 26 instruction-tuned models. To assess the results, we employ both human evaluation and LLM-as-a-judge methods. Human evaluation scores showed moderate to high correlation with benchmark evaluations, depending on the dataset. Claude 3.7 Sonnet, used as an LLM judge, demonstrated strong alignment with human ratings, indicating that top-performing LLMs can effectively support the evaluation of Estonian-language models.
Paper Structure (14 sections, 3 figures, 12 tables)

This paper contains 14 sections, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Types of LM benchmarks
  4. Benchmarking for underrepresented languages
  5. Benchmarks
  6. Results
  7. Base Models
  8. Instruction-Tuned Models
  9. Correlations between tasks
  10. Impact of model scale
  11. Validation
  12. Human Evaluation
  13. LLM-as-a-Judge
  14. Conclusion

Figures (3)

  • Figure 1: A game card from the quiz boardgame that was the source for the Trivia dataset.
  • Figure 2: Pearson correlation matrix between the results of different datasets.
  • Figure 3: Model size vs. mean score (open models only). Absolute parameter size was used.