Building Russian Benchmark for Evaluation of Information Retrieval Models

TL;DR

RusBEIR presents a BEIR-inspired framework for zero-shot evaluation of Russian information retrieval models, aggregating 17 datasets from translated, multilingual, existing Russian, and newly created Wikipedia-based sources. It emphasizes careful preprocessing for lexical methods and demonstrates that neural models (e.g., mE5-large, BGE-M3) generally outperform lexical baselines on most datasets, though long-document retrieval is hampered by input-size constraints. The benchmark is BEIR-compatible and open-source, enabling fair cross-dataset comparisons and reproducibility, while its Wikipedia-based datasets enable systematic study of document length effects. Overall, the work highlights the practical viability of BM25 as a strong baseline for full-document retrieval and maps the strengths and limitations of neural IR models in Russian, guiding future efficiency and length-robust retrieval research.

Abstract

We introduce RusBEIR, a comprehensive benchmark designed for zero-shot evaluation of information retrieval (IR) models in the Russian language. Comprising 17 datasets from various domains, it integrates adapted, translated, and newly created datasets, enabling systematic comparison of lexical and neural models. Our study highlights the importance of preprocessing for lexical models in morphologically rich languages and confirms BM25 as a strong baseline for full-document retrieval. Neural models, such as mE5-large and BGE-M3, demonstrate superior performance on most datasets, but face challenges with long-document retrieval due to input size constraints. RusBEIR offers a unified, open-source framework that promotes research in Russian-language information retrieval.