Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQ

Carolin Holtermann, Paul Röttger, Timm Dill, Anne Lauscher

TL;DR

MultiQ tackles the persistent underrepresentation of non-English languages in open LLMs by introducing a silver-standard benchmark of 27,400 open-ended questions across 137 languages to probe two dimensions: language fidelity (prompt-language responses) and QA accuracy (correctness of answers). Built via a two-step dataset creation process (100 LMSYS questions and 100 GPT-4 questions) translated into 136 languages, MultiQ is validated by native annotators and ensures broad typological diversity (137 languages, 20 families, 95.4% Grambank feature coverage). Evaluations across six open LLMs reveal substantial variability: some models faithful to the prompt language still underperform in accuracy, while others achieve higher accuracy with English responses; overall, English questions are the easiest and some languages exhibit a long tail of poor performance. Tokenization emerges as a key explanatory factor: subword tokenization correlates with higher QA accuracy, and languages that are tokenized into subwords fare better than those relying on characters or ASCII tokens. Overall, MultiQ provides a scalable, typologically diverse framework to guide improvements in multilingual capabilities of open LLMs, especially for underserved languages, while cautioning that high performance on this benchmark does not imply readiness for language-specific deployment.

Abstract

Large language models (LLMs) need to serve everyone, including a global majority of non-English speakers. However, most LLMs today, and open LLMs in particular, are often intended for use in just English (e.g. Llama2, Mistral) or a small handful of high-resource languages (e.g. Mixtral, Qwen). Recent research shows that, despite limits in their intended use, people prompt LLMs in many different languages. Therefore, in this paper, we investigate the basic multilingual capabilities of state-of-the-art open LLMs beyond their intended use. For this purpose, we introduce MultiQ, a new silver standard benchmark for basic open-ended question answering with 27.4k test questions across a typologically diverse set of 137 languages. With MultiQ, we evaluate language fidelity, i.e. whether models respond in the prompted language, and question answering accuracy. All LLMs we test respond faithfully and/or accurately for at least some languages beyond their intended use. Most models are more accurate when they respond faithfully. However, differences across models are large, and there is a long tail of languages where models are neither accurate nor faithful. We explore differences in tokenization as a potential explanation for our findings, identifying possible correlations that warrant further investigation.

Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQ

TL;DR

MultiQ tackles the persistent underrepresentation of non-English languages in open LLMs by introducing a silver-standard benchmark of 27,400 open-ended questions across 137 languages to probe two dimensions: language fidelity (prompt-language responses) and QA accuracy (correctness of answers). Built via a two-step dataset creation process (100 LMSYS questions and 100 GPT-4 questions) translated into 136 languages, MultiQ is validated by native annotators and ensures broad typological diversity (137 languages, 20 families, 95.4% Grambank feature coverage). Evaluations across six open LLMs reveal substantial variability: some models faithful to the prompt language still underperform in accuracy, while others achieve higher accuracy with English responses; overall, English questions are the easiest and some languages exhibit a long tail of poor performance. Tokenization emerges as a key explanatory factor: subword tokenization correlates with higher QA accuracy, and languages that are tokenized into subwords fare better than those relying on characters or ASCII tokens. Overall, MultiQ provides a scalable, typologically diverse framework to guide improvements in multilingual capabilities of open LLMs, especially for underserved languages, while cautioning that high performance on this benchmark does not imply readiness for language-specific deployment.

Abstract

Large language models (LLMs) need to serve everyone, including a global majority of non-English speakers. However, most LLMs today, and open LLMs in particular, are often intended for use in just English (e.g. Llama2, Mistral) or a small handful of high-resource languages (e.g. Mixtral, Qwen). Recent research shows that, despite limits in their intended use, people prompt LLMs in many different languages. Therefore, in this paper, we investigate the basic multilingual capabilities of state-of-the-art open LLMs beyond their intended use. For this purpose, we introduce MultiQ, a new silver standard benchmark for basic open-ended question answering with 27.4k test questions across a typologically diverse set of 137 languages. With MultiQ, we evaluate language fidelity, i.e. whether models respond in the prompted language, and question answering accuracy. All LLMs we test respond faithfully and/or accurately for at least some languages beyond their intended use. Most models are more accurate when they respond faithfully. However, differences across models are large, and there is a long tail of languages where models are neither accurate nor faithful. We explore differences in tokenization as a potential explanation for our findings, identifying possible correlations that warrant further investigation.
Paper Structure (31 sections, 11 figures, 11 tables)

This paper contains 31 sections, 11 figures, 11 tables.

Table of Contents

  1. Introduction
  2. The MultiQ Dataset
  3. Dataset Creation
  4. Validation of Translations
  5. Typological Diversity
  6. Experiments and Results
  7. Overall Experimental Setup
  8. Models
  9. Inference
  10. Language Fidelity
  11. Question Answering Accuracy
  12. Automated Evaluation
  13. Results
  14. Language Fidelity vs Answer Accuracy
  15. Tokenization and Multilinguality
  16. ...and 16 more sections

Figures (11)

  • Figure 1: The 137 languages covered in our MultiQ question dataset. We show their geographic location according to the WALS database and indicate their corresponding language family through colors.
  • Figure 2: Distributions of the pairwise lang2vec distances for each language pair present in MultiQ.
  • Figure 3: Overall language fidelity. Proportion of model responses (%) in the same language as the input prompt, in English, or in another language. We evaluate the responses of six models for 200 prompts in 135 languages (excl. Dogri & Meiteilon).
  • Figure 4: Granular language fidelity. Correlation matrices illustrating the relationship between input prompt and model response languages, shown as percentages. Axis ticks are selectively labeled for better visualization.
  • Figure 5: Answer accuracy on MultiQ in proportion (%) of correctly answered questions per language. We compare four models of the same size across 137 languages and sort the results by median accuracy.
  • ...and 6 more figures