One ruler to measure them all: Benchmarking multilingual long-context language models

TL;DR

OneRuler presents a multilingual long-context benchmark that extends the English-centric Ruler by testing retrieval and aggregation across 26 languages with seven synthetic tasks (five NIAH variants and two CWE aggregations) at context lengths up to 128K. The dataset is built via English task definitions translated by native speakers, including a novel None option to handle nonexistent needles, and evaluated on open- and closed-weight LLMs, revealing a widening language-resource gap and surprising language rankings (e.g., Polish leading, English not top). The results show CWE aggregation is substantially harder than NIAH, and cross-lingual instruction-context configurations significantly impact performance. The work also highlights tokenizer and output-token challenges in multilingual long-context settings, and it releases ONERULER to drive future advances in multilingual long-context training and evaluation.

Abstract

We present ONERULER, a multilingual benchmark designed to evaluate long-context language models across 26 languages. ONERULER adapts the English-only RULER benchmark (Hsieh et al., 2024) by including seven synthetic tasks that test both retrieval and aggregation, including new variations of the "needle-in-a-haystack" task that allow for the possibility of a nonexistent needle. We create ONERULER through a two-step process, first writing English instructions for each task and then collaborating with native speakers to translate them into 25 additional languages. Experiments with both open-weight and closed LLMs reveal a widening performance gap between low- and high-resource languages as context length increases from 8K to 128K tokens. Surprisingly, English is not the top-performing language on long-context tasks (ranked 6th out of 26), with Polish emerging as the top language. Our experiments also show that many LLMs (particularly OpenAI's o3-mini-high) incorrectly predict the absence of an answer, even in high-resource languages. Finally, in cross-lingual scenarios where instructions and context appear in different languages, performance can fluctuate by up to 20% depending on the instruction language. We hope the release of ONERULER will facilitate future research into improving multilingual and cross-lingual long-context training pipelines.

Figures (25)

• Figure 1: (A) Micro-accuracy of all models on the S-NIAH task for the top 5 and bottom 5 languages by Wikipedia size. As context length increases, the performance gap between high-resource and low-resource languages increases. (B) Performance of o3-mini-high on the S-NIAH task in English, with and without the inclusion of the "None" option that allows for the possibility of a nonexistent needle. Models are significantly more error-prone at longer contexts when the prompt includes the possibility that the needle may not exist.
• Figure 2: The seven tasks included in OneRuler. Spans highlighted in red are distractors, while green spans contain answers that need to be produced for credit. CWE appears twice (in easy and hard versions with differing frequencies) but shares the same format, hence only one version is shown here. The None-NIAH task is a novel variant in which the needle does not exist in the input context.
• Figure 3: Micro-accuracy across context-lengths and languages for all NIAH tasks. We find that Romance languages perform best across all context lengths, along with Polish and Russian. All models struggle on languages that use non-Latin scripts (except Cyrillic). Gemini-1.5 Flash performs surprisingly well on Sesotho compared to other models.
• Figure 4: NIAH performance across models and languages by language resource group for long-context tasks (64K and 128K). Gemini 1.5 Flash demonstrates the best long-context performance, while English and Chinese are surprisingly not among the top five languages.
• Figure 5: The performance of models on each task, with bars representing English, all other high-resource languages, and low-resource languages.