IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation

TL;DR

IRLBench addresses the lack of robust multilingual evaluation for extremely low-resource languages by providing a parallel Irish–English, multimodal dataset derived from the 2024 Leaving Certificate exams. The authors propose an open-ended generation task evaluated via an LLM-as-a-judge framework, incorporating official marking schemes and a language-fidelity check with FastText. Across closed- and open-source models, a clear English–Irish performance gap is observed, with English accuracy around $76.2\%$ and Irish around $55.8\%$, and Irish-language outputs being produced in less than $80\%$ of cases. The work highlights challenges in multilingual transfer and language grounding, and releases dataset and code to spur future work on culturally aware multilingual AI.

Abstract

Recent advances in Large Language Models (LLMs) have demonstrated promising knowledge and reasoning abilities, yet their performance in multilingual and low-resource settings remains underexplored. Existing benchmarks often exhibit cultural bias, restrict evaluation to text-only, rely on multiple-choice formats, and, more importantly, are limited for extremely low-resource languages. To address these gaps, we introduce IRLBench, presented in parallel English and Irish, which is considered definitely endangered by UNESCO. Our benchmark consists of 12 representative subjects developed from the 2024 Irish Leaving Certificate exams, enabling fine-grained analysis of model capabilities across domains. By framing the task as long-form generation and leveraging the official marking scheme, it does not only support a comprehensive evaluation of correctness but also language fidelity. Our extensive experiments of leading closed-source and open-source LLMs reveal a persistent performance gap between English and Irish, in which models produce valid Irish responses less than 80\% of the time, and answer correctly 55.8\% of the time compared to 76.2\% in English for the best-performing model. We release IRLBench (https://huggingface.co/datasets/ReliableAI/IRLBench) and an accompanying evaluation codebase (https://github.com/ReML-AI/IRLBench) to enable future research on robust, culturally aware multilingual AI development.

Figures (7)

• Figure 1: Performance of State-of-the-Art models on recent Irish benchmarks compared to IRLBench. We benchmark more models on IRLBench in Section \ref{['sec:exp']}.
• Figure 2: Illustrative examples showcasing the Irish-English parallel nature and the diverse tasks of IRLBench dataset.
• Figure 3: Pipeline.
• Figure 4: Distributions of subject groups and subjects as percentages of total amount of samples in IRLBench.
• Figure 5: Accuracy scores on IRLBench per model and language.