MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries

TL;DR

MiLQ introduces the first public mixed-language query benchmark (MQIR) crafted by bilingual users to study bilingual web search. It analyzes realism, user preference for mixed-language formulations, and the performance of multilingual IR models—including ColBERT-based retrievers trained via a Translate-Distill approach—across native, English, and mixed queries; it also examines the impact of test-time translation on retrieval. Key findings show moderate MQIR performance with inconsistent results across query types, and that intentionally mixing English terms can boost English-document retrieval through improved token matching. The work points to the potential of code-switched training data for building more robust bilingual IR systems and highlights directions toward larger MQIR benchmarks and multilingual LLM-driven approaches.

Abstract

Despite bilingual speakers frequently using mixed-language queries in web searches, Information Retrieval (IR) research on them remains scarce. To address this, we introduce MiLQ, Mixed-Language Query test set, the first public benchmark of mixed-language queries, qualified as realistic and relatively preferred. Experiments show that multilingual IR models perform moderately on MiLQ and inconsistently across native, English, and mixed-language queries, also suggesting code-switched training data's potential for robust IR models handling such queries. Meanwhile, intentional English mixing in queries proves an effective strategy for bilinguals searching English documents, which our analysis attributes to enhanced token matching compared to native queries.

Figures (9)

• Figure 1: Illustration of a bilingual user freely using German, English, and mixed-language queries. German elements are in rgb]0.7,0.9,0.7green, and English in rgb]1.0,0.8,0.7orange.
• Figure 2: Performance of retrieval models across rgb]1,0.4,0CLIR, rgb]1,0.8,0MQIR (MiLQ), and rgb]0.4,0.6,1MonoIR scenarios. Results are averaged by language group: low-resource (SW, SO; MAP@100) [left], high-resource (FI, DE, FR; MAP@100) [middle], and diverse document language (ZH, FA, RU; nDCG@20) [right]. Models include BM25, specialized multi-vector dense retrievers (Mono-, Mixed-, Cross-Distill), and mContriever. See Appendix \ref{['sec:appendix_individual_performance']} for per-language details.
• Figure 3: Token-level similarity matrices from Cross-Distill for German and mixed-language queries on ground truth passage. The y-axis shows tokenized queries (mixed-language left, native right), and the x-axis represents the tokenized English passage. MaxSim tokens are marked by $\times$, and the code-switched parts are rgb]1.0, 1.0, 0.3highlighted.
• Figure 4: Guideline for German-English mixed-language search query annotators.
• Figure 5: Examples of Title queries from the MiLQ dataset. Code-switched segments are highlighted, and CMI values are shown in parentheses. (*Note: Although 'Catastrophe' is also a French word, it was identified as English by the language model in this instance.)