Hindi-BEIR : A Large Scale Retrieval Benchmark in Hindi

TL;DR

The paper tackles the absence of a large-scale, diverse Hindi information retrieval benchmark by introducing Hindi-BEIR, a BEIR-like suite spanning 15 datasets and 8 tasks across multiple domains. It constructs the benchmark via three pillars: translating high-quality English BEIR data using IndicTrans2 with back-translation quality checks, creating cross-lingual Hindi CC News retrieval data, and compiling existing multilingual datasets (e.g., MIRACL, MLDR, mMARCO, IndicQARetrieval, WikiPediaRetrieval). Core contributions include the benchmark construction, public data release, and empirical evaluation of state-of-the-art multilingual retrievers (e.g., mE5, BGE-M3) versus embedding-based baselines (e.g., LASER, LaBSE), highlighting the benefits of task-specific fine-tuning and the impact of domain and document length. The work provides a standardized resource to accelerate Hindi and Indic-language retrieval research and guides future improvements in cross-lingual, long-document, and domain-adaptive retrieval approaches.

Abstract

Given the large number of Hindi speakers worldwide, there is a pressing need for robust and efficient information retrieval systems for Hindi. Despite ongoing research, there is a lack of comprehensive benchmark for evaluating retrieval models in Hindi. To address this gap, we introduce the Hindi version of the BEIR benchmark, which includes a subset of English BEIR datasets translated to Hindi, existing Hindi retrieval datasets, and synthetically created datasets for retrieval. The benchmark is comprised of $15$ datasets spanning across $8$ distinct tasks. We evaluate state-of-the-art multilingual retrieval models on this benchmark to identify task and domain-specific challenges and their impact on retrieval performance. By releasing this benchmark and a set of relevant baselines, we enable researchers to understand the limitations and capabilities of current Hindi retrieval models, promoting advancements in this critical area. The datasets from Hindi-BEIR are publicly available.

Figures (45)

• Figure 1: An example of a query with its corresponding golden corpus from the ArguAna Dataset
• Figure 2: Distribution of the number of words in the queries of ArguAna Dataset
• Figure 3: Distribution of Number of Words in corpus of ArguAna Dataset
• Figure 4: An example of a query with its corresponding golden corpus from the FiQA-2018 Dataset
• Figure 5: Distribution of the number of words in the queries of FiQA-2018 Dataset