What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models

TL;DR

The paper addresses CLIR by disentangling translation-based pipelines from semantic multilingual retrieval. It systematically compares five multilingual encoders, four interventions (document translation, contrastive learning at word/phrase/QD levels, and cross-encoder re-ranking), and ANN vs. exact search across three benchmarks, evaluating with $Recall@100$ and $nDCG@100$ metrics. The findings show that dense, semantically aligned multilingual embeddings outperform translation-based approaches, with contrastive learning markedly helping encoders with weak initial alignment and re-ranking offering variable gains depending on data quality; ANN yields limited efficiency gains at the tested scales. Practically, the work advocates prioritizing semantic multilingual embeddings and targeted alignment over translation-heavy pipelines, especially for cross-script and under-resourced language pairs, while cautioning that efficiency tools like ANN may only pay off at much larger scales. The results advance understanding of how linguistic factors, model alignment, and training signals shape cross-lingual retrieval performance and offer guidance for building scalable, accurate CLIR systems.

Abstract

Cross-lingual information retrieval (CLIR) enables access to multilingual knowledge but remains challenging due to disparities in resources, scripts, and weak cross-lingual semantic alignment in embedding models. Existing pipelines often rely on translation and monolingual retrieval heuristics, which add computational overhead and noise, degrading performance. This work systematically evaluates four intervention types, namely document translation, multilingual dense retrieval with pretrained encoders, contrastive learning at word, phrase, and query-document levels, and cross-encoder re-ranking, across three benchmark datasets. We find that dense retrieval models trained specifically for CLIR consistently outperform lexical matching methods and derive little benefit from document translation. Contrastive learning mitigates language biases and yields substantial improvements for encoders with weak initial alignment, and re-ranking can be effective, but depends on the quality of the cross-encoder training data. Although high-resource languages still dominate overall performance, gains over lexical and document-translated baselines are most pronounced for low-resource and cross-script pairs. These findings indicate that cross-lingual search systems should prioritise semantic multilingual embeddings and targeted learning-based alignment over translation-based pipelines, particularly for cross-script and under-resourced languages.

Figures (8)

• Figure 1: Overview of the CLIR ranking evaluation pipeline.
• Figure 2: Translation quality of different models evaluated with COMET and Perplexity, by translating non-English documents from the CLIRMatrix dataset sun-duh-2020-clirmatrix, sampling the same 100 documents from each language.
• Figure 3: Comparison of lexical BM25 retrieval (Recall@100) on query–document language pairs for cross-lingual retrieval (a) without and (b) with document translation on the mMARCO dataset bonifacio_2022_mmarco.
• Figure 4: Performance (Recall@100) of five pretrained multilingual encoders using cosine similarity between query and document embeddings without translation. Results are averaged across all language pairs and shown for selected pairs.
• Figure 5: Performance of the best-performing embedding-based model on each dataset and BM25, with and without document translation, on average, and across selected query–document language pairs.