Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting

TL;DR

Irish-BLiMP introduces a linguistically informed benchmark for evaluating Irish grammar by constructing 1020 minimal pairs across 11 phenomena and validating them with fluent speakers. The study assesses a broad range of open- and closed-source LLMs against native Irish baselines using a discriminative multiple-choice framework, revealing a substantial gap: humans average $90.09\%$ accuracy while best models peak around $81.1\%$ under few-shot prompting and much lower in zero-shot, with open models nearing chance. The results show a weak alignment between model and human errors, evidence that current systems rely more on surface patterns than internalized Irish grammar, and a notable open-vs-closed performance gap of about $18.1$ percentage points. The work establishes Irish-BLiMP as a benchmark for advancing grammatical understanding in extremely low-resource languages and motivates future expansion to dialects and richer linguistic coverage.

Abstract

We present Irish-BLiMP (Irish Benchmark of Linguistic Minimal Pairs), the first dataset and framework designed for fine-grained evaluation of linguistic competence in the Irish language, an endangered language. Drawing on a variety of linguistic literature and grammar reference works, we manually constructed and reviewed 1020 minimal pairs across a taxonomy of 11 linguistic features, through a team of fluent Irish speakers. We evaluate both existing Large Language Models (LLMs) and fluent human participants on their syntactic knowledge of Irish. Our findings show that humans outperform all models across all linguistic features, achieving 16.6% higher accuracy on average. Moreover, a substantial performance gap of 18.1% persists between open- and closed-source LLMs, with even the strongest model (gpt-5) reaching only 73.5% accuracy compared to 90.1% by human. Interestingly, human participants and models struggle on different aspects of Irish grammar, thus highlighting a difference in representation learned by the models. Overall, Irish-BLiMP provides the first systematic framework for evaluating the grammatical competence of LLMs in Irish and offers a valuable benchmark for advancing research on linguistic understanding in low-resource languages.

Figures (4)

• Figure 1: Overview of the data creation process for the Irish minimal pairs dataset, from source identification to final dataset assembly.
• Figure 2: Zero-shot accuracy (%) by model and category.
• Figure 3: Correlation between models across categories.
• Figure 4: Accuracy by model (groups) and prompting technique (bars).