GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models

TL;DR

GlotEval addresses the lack of scalable multilingual evaluation for large language models, particularly in low-resource languages, by integrating 27 benchmarks and standardizing language codes via ISO 639-3. It introduces language-specific prompt templates and a multilingual prompt builder to enable non-English-centered evaluation, while supporting efficient inference through backends like vLLM and HF Transformers. A multilingual translation case study demonstrates that language-centric prompting and cross-language evaluation reveal strengths and weaknesses across languages, guiding improvements in instruction-following and translation quality. The framework emphasizes transparency and scalability, offering per-sample data export for error analysis and future enhancement, including more benchmarks and human-in-the-loop evaluation. Overall, GlotEval provides a scalable, inclusive platform for diagnosing multilingual LLM performance across a broad linguistic spectrum.

Abstract

Large language models (LLMs) are advancing at an unprecedented pace globally, with regions increasingly adopting these models for applications in their primary language. Evaluation of these models in diverse linguistic environments, especially in low-resource languages, has become a major challenge for academia and industry. Existing evaluation frameworks are disproportionately focused on English and a handful of high-resource languages, thereby overlooking the realistic performance of LLMs in multilingual and lower-resource scenarios. To address this gap, we introduce GlotEval, a lightweight framework designed for massively multilingual evaluation. Supporting seven key tasks (machine translation, text classification, summarization, open-ended generation, reading comprehension, sequence labeling, and intrinsic evaluation), spanning over dozens to hundreds of languages, GlotEval highlights consistent multilingual benchmarking, language-specific prompt templates, and non-English-centric machine translation. This enables a precise diagnosis of model strengths and weaknesses in diverse linguistic contexts. A multilingual translation case study demonstrates GlotEval's applicability for multilingual and language-specific evaluations.

Figures (4)

• Figure 1: Workflow of GlotEval
• Figure 2: GlotEval benchmark data loader
• Figure 3: Benchmark data loader components: (a) Language ID alignment process and (b) multilingual prompt generation.
• Figure 4: ChrF scores for different translation directions comparing EMMA-500 and Llama-2-7B across four prompting strategies.