The Overlooked Role of Graded Relevance Thresholds in Multilingual Dense Retrieval

TL;DR

This work investigates how graded relevance signals and the threshold used to binarize them affect multilingual dense retrieval. Using an LLM-based pipeline to craft a synthetic, graded-relevance dataset across six languages, the authors evaluate monolingual, mixed-language, and cross-lingual retrieval with a baseline multilingual retriever, examining how the threshold $\tau$ impacts performance and data efficiency. They find that the optimal $\tau$ is not universal; lower-resource languages benefit from more inclusive thresholds while higher-resource languages prefer stricter thresholds, and language mixing further modulates optimal calibration. The study highlights threshold calibration as a central design choice in multilingual dense retrieval, capable of improving effectiveness, mitigating annotation noise, and reducing training data requirements, while also noting cross-annotator variability and pre-training influences. The results advocate for principled, language- and task-aware threshold setting as a practical lever to harness graded relevance in real-world multilingual IR systems.

Abstract

Dense retrieval models are typically fine-tuned with contrastive learning objectives that require binary relevance judgments, even though relevance is inherently graded. We analyze how graded relevance scores and the threshold used to convert them into binary labels affect multilingual dense retrieval. Using a multilingual dataset with LLM-annotated relevance scores, we examine monolingual, multilingual mixture, and cross-lingual retrieval scenarios. Our findings show that the optimal threshold varies systematically across languages and tasks, often reflecting differences in resource level. A well-chosen threshold can improve effectiveness, reduce the amount of fine-tuning data required, and mitigate annotation noise, whereas a poorly chosen one can degrade performance. We argue that graded relevance is a valuable but underutilized signal for dense retrieval, and that threshold calibration should be treated as a principled component of the fine-tuning pipeline.

Figures (7)

• Figure 1: Dense retrieval models are predominantly trained with contrastive learning. Annotating data with graded relevance scores, rather than binary labels, enables selecting an appropriate threshold for positive and negative pairs. This threshold choice has a strong impact on contrastive fine-tuning performance, particularly in multilingual settings.
• Figure 2: Number of documents (Wikipedia articles) per language in the MIRACL dataset. In our analyses we use Finnish (fi), Arabic (ar), Japanese (ja), Russian (ru), Spanish (es) and English.
• Figure 3: Distributions of relevance scores across six languages, as provided by the two LLM annotators (Gpt-4o and Gemma-3-27B). The relevance score ranges from 0 (passage not relevant to query) to 3 (passage very relevant to query). The main difference is apparant in labels 2 and 3.
• Figure 4: Agreement heatmaps between the two LLM annotators (GPT-4o and Gemma-3-27B) across languages. Each cell shows the row-normalized frequency of Gemma-3-27B’s scores conditioned on GPT-4o’s score. High-resource languages show stronger diagonal agreement (higher Quadratic Weighted $\kappa$), while low-resource languages exhibit greater off-diagonal noise (lower $\kappa$). These differences in annotation consistency further motivate examining how threshold choices affect the preparation of finetuning data.
• Figure 5: Monolingual retrieval performance (nDCG@10) of me5 on MIRACL (horizontal dashed lines) and after fine-tuning, across six languages. Each subplot shows performance as a function of training set size under different relevance thresholds used to convert graded LLM scores into contrastive training pairs. Lower-resource languages tend to benefit from lower thresholds, whereas higher-resource languages perform better with higher thresholds.