Wikipedia-based Datasets in Russian Information Retrieval Benchmark RusBEIR
Grigory Kovalev, Natalia Loukachevitch, Mikhail Tikhomirov, Olga Babina, Pavel Mamaev
TL;DR
This paper introduces the Wikipedia Interesting Facts series, a set of Russian IR datasets derived from the Russian Wikipedia 'Did you know...' section to broaden the RusBEIR benchmark. It systematically evaluates lexical (BM25) and neural retrievers, including diverse dense models and neural rerankers, across tasks from full-document retrieval to fact-checking and retrieval-augmented generation. Key findings show lexical methods excel on long texts, neural approaches shine on shorter chunks, and combining retrieval with neural reranking yields substantial gains, especially in larger corpora. The work provides a scalable methodology, expands public resources on HuggingFace and GitHub, and highlights the practical value of language-specific models and reranking in Russian IR research.
Abstract
In this paper, we present a novel series of Russian information retrieval datasets constructed from the "Did you know..." section of Russian Wikipedia. Our datasets support a range of retrieval tasks, including fact-checking, retrieval-augmented generation, and full-document retrieval, by leveraging interesting facts and their referenced Wikipedia articles annotated at the sentence level with graded relevance. We describe the methodology for dataset creation that enables the expansion of existing Russian Information Retrieval (IR) resources. Through extensive experiments, we extend the RusBEIR research by comparing lexical retrieval models, such as BM25, with state-of-the-art neural architectures fine-tuned for Russian, as well as multilingual models. Results of our experiments show that lexical methods tend to outperform neural models on full-document retrieval, while neural approaches better capture lexical semantics in shorter texts, such as in fact-checking or fine-grained retrieval. Using our newly created datasets, we also analyze the impact of document length on retrieval performance and demonstrate that combining retrieval with neural reranking consistently improves results. Our contribution expands the resources available for Russian information retrieval research and highlights the importance of accurate evaluation of retrieval models to achieve optimal performance. All datasets are publicly available at HuggingFace. To facilitate reproducibility and future research, we also release the full implementation on GitHub.