Know When to Fuse: Investigating Non-English Hybrid Retrieval in the Legal Domain

TL;DR

This work studies the efficacy of hybrid search across a variety of prominent retrieval models within the unexplored field of law in the French language, assessing both zero-shot and in-domain scenarios.

Abstract

Hybrid search has emerged as an effective strategy to offset the limitations of different matching paradigms, especially in out-of-domain contexts where notable improvements in retrieval quality have been observed. However, existing research predominantly focuses on a limited set of retrieval methods, evaluated in pairs on domain-general datasets exclusively in English. In this work, we study the efficacy of hybrid search across a variety of prominent retrieval models within the unexplored field of law in the French language, assessing both zero-shot and in-domain scenarios. Our findings reveal that in a zero-shot context, fusing different domain-general models consistently enhances performance compared to using a standalone model, regardless of the fusion method. Surprisingly, when models are trained in-domain, we find that fusion generally diminishes performance relative to using the best single system, unless fusing scores with carefully tuned weights. These novel insights, among others, expand the applicability of prior findings across a new field and language, and contribute to a deeper understanding of hybrid search in non-English specialized domains.

Figures (11)

• Figure 1: A high-level illustration of the hybrid search workflow based on various sparse and dense retrievers.
• Figure 2: In-domain score distributions of domain-general end-to-end retrievers, normalized using min-max, z-score, and percentile scaling. The distributions are derived from ranking all 27,942 articles in LLeQA's knowledge corpus against the 201 development set queries, resulting in approximately 5.6 million scores per system.
• Figure 3: Illustration of the complementary relationship between a sparse (BM25) and a dense (ColBERT$_{\textsc{fr-base}}$) system on out-of-distribution data. Scores have been min-max normalized and categorized into four distinct regions based on each system's global distribution, depicted in \ref{['fig:normalized_distributions']}.
• Figure 4: Effect of weight tuning in normalized score fusion between BM25 and DPR$_{\textsc{fr-\{lex,base\}}}$ on LLeQA dev set.
• Figure 5: High-level illustration of the four prominent neural retrieval architectures explored in this study.