Enabling Low-Resource Language Retrieval: Establishing Baselines for Urdu MS MARCO

TL;DR

This work tackles data scarcity for Urdu information retrieval by translating the English MS MARCO dataset into Urdu using IndicTrans2 and establishing strong baselines with BM25 and a zero-shot re-ranker. It then fine-tunes the mMARCO model on the Urdu-translated dataset (Urdu-mT5-mMARCO), achieving a MRR@10 of 0.248 and Recall@10 of 0.438, surpassing zero-shot performance and demonstrating large gains from language-specific fine-tuning. The study highlights translation quality as a crucial factor influencing retrieval outcomes and publicly releases the Urdu MS MARCO dataset, the fine-tuned model, and code to foster further research in Urdu and other low-resource languages. Overall, the approach provides a scalable pathway to expand multilingual IR capabilities, reducing access gaps for Urdu speakers and informing future work in South Asian languages.

Abstract

As the Information Retrieval (IR) field increasingly recognizes the importance of inclusivity, addressing the needs of low-resource languages remains a significant challenge. This paper introduces the first large-scale Urdu IR dataset, created by translating the MS MARCO dataset through machine translation. We establish baseline results through zero-shot learning for IR in Urdu and subsequently apply the mMARCO multilingual IR methodology to this newly translated dataset. Our findings demonstrate that the fine-tuned model (Urdu-mT5-mMARCO) achieves a Mean Reciprocal Rank (MRR@10) of 0.247 and a Recall@10 of 0.439, representing significant improvements over zero-shot results and showing the potential for expanding IR access for Urdu speakers. By bridging access gaps for speakers of low-resource languages, this work not only advances multilingual IR research but also emphasizes the ethical and societal importance of inclusive IR technologies. This work provides valuable insights into the challenges and solutions for improving language representation and lays the groundwork for future research, especially in South Asian languages, which can benefit from the adaptable methods used in this study.