Linguini: A benchmark for language-agnostic linguistic reasoning

TL;DR

Linguini tackles the problem of evaluating linguistic reasoning without language-specific priors by curating a Rosetta Stone–style benchmark from the International Linguistic Olympiad. The authors construct Linguini with 894 questions across 75 low-resource languages, categorized into sequence transduction, fill-in-blanks, and digit/text number transliteration, and evaluate both open and proprietary models under zero- to five-shot in-context settings with careful contamination controls. Key findings include a persistent gap between best proprietary (24.05%) and best open models (8.84%), strong reliance on contextual information, and evidence that non-Latin-script representations can be leveraged without losing reasoning capability. The work demonstrates the benchmark’s usefulness for measuring linguistic reasoning beyond language-specific knowledge and discusses limitations, scalability, and potential for using textbooks as in-context learning sources to further probe model capabilities and generalization across languages.

Abstract

We propose a new benchmark to measure a language model's linguistic reasoning skills without relying on pre-existing language-specific knowledge. The test covers 894 questions grouped in 160 problems across 75 (mostly) extremely low-resource languages, extracted from the International Linguistic Olympiad corpus. To attain high accuracy on this benchmark, models don't need previous knowledge of the tested language, as all the information needed to solve the linguistic puzzle is presented in the context. We find that, while all analyzed models rank below 25% accuracy, there is a significant gap between open and closed models, with the best-performing proprietary model at 24.05% and the best-performing open model at 8.84%.

Figures (6)

• Figure 1: Examples of Linguini entries covering the three problems included in the dataset: sequence transduction, fill-in-blanks, number transliteration.
• Figure 2: A subset of the context of a problem in Terenâ language and the reasoning steps needed to solve it. To correctly answer the question, the model must notice that (a) voiced d mutates to voiceless paired sound t (fortition), (b) n is dropped because there are no voiceless nasal alveolar sounds and (c) an epenthetic vowel has to be added between the mutation consonant and the rest of the word (a root), and that the vowel that gets added matches the aperture of the vowel in the root. If the aperture is closed, the epenthetic vowel is the closed front vowel i; if the aperture is mid, the epenthetic vowel is the mid front vowel e.
• Figure 3: Example of transliteration of a problem into Cyrillic, Greek, Georgian and Armenian scripts.
• Figure 4: Accuracy vs. number of speakers. Data points are clustered for readability.
• Figure 5: Accuracy vs. number of Google searches. Data points are clustered for readability.