Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages

TL;DR

This study asks whether large language models hold explicit metalinguistic knowledge across the world's languages. It introduces a massively multilingual benchmark derived from the World Atlas of Language Structures (WALS), transforming 192 features into language-specific multiple-choice questions and evaluating GPT-4o and two open models across 2,660 languages. Results show metalinguistic knowledge is limited and unevenly distributed, with accuracy about 0.37 for the best model and strongly tied to data availability rather than universal grammatical competence; language resources and typological proximity to English further modulate performance. The authors provide an open-source benchmark to promote inclusive, cross-linguistic evaluation and advocate for more diverse, resource-rich data to close observed gaps in metalinguistic knowledge.

Abstract

Large language models (LLMs) are routinely evaluated on language use tasks, yet their knowledge of linguistic structure remains poorly understood. Existing linguistic benchmarks typically focus on narrow phenomena, emphasize high-resource languages, and rarely evaluate metalinguistic knowledge-explicit reasoning about language structure rather than language use. Using accuracy and macro F1, together with majority-class and chance baselines, we analyse overall performance and examine variation by linguistic domains and language-related factors. Our results show that metalinguistic knowledge in current LLMs is limited: GPT-4o performs best but achieves only moderate accuracy (0.367), while open-source models lag behind. All models perform above chance but fail to outperform the majority-class baseline, suggesting they capture cross-linguistic patterns but lack fine-grained grammatical distinctions. Performance varies across linguistic domains, with lexical features showing the highest accuracy and phonological features among the lowest, partially reflecting differences in online visibility. At the language level, accuracy shows a strong association with digital language status: languages with higher digital presence and resource availability are evaluated more accurately, while low-resource languages show substantially lower performance. Analyses of predictive factors confirm that resource-related indicators (Wikipedia size, corpus availability) are more informative predictors of accuracy than geographical, genealogical, or sociolinguistic factors. Together, these results suggest that LLMs' metalinguistic knowledge is fragmented and shaped by data availability rather than generalizable grammatical competence across the world's languages. We release our benchmark as an open-source dataset to support systematic evaluation and encourage greater global linguistic diversity in future LLMs.

Figures (8)

• Figure 1: High-level overview of our evaluation setup. While LLMs can use grammatical patterns correctly in language generation (top example), we assess their explicit linguistic knowledge by querying models with WALS-based multiple-choice questions and comparing their responses to the corresponding ground-truth feature values documented in WALS (bottom example).
• Figure 2: A feature page from WALS Online illustrating how each feature defines a set of possible values (right panel) and maps their distribution across languages (bottom panel).
• Figure 3: Normalized LLM accuracy across linguistic domains relative to majority-class and chance baselines, ranked by GPT-4o performance.
• Figure 4: Correlation between domain-level accuracy and online visibility (mean Google hits per domain) for GPT-4o (r = 0.715).
• Figure 5: Distribution of accuracy by digital status (0 = very low to 5 = dominant) for GPT-4o.