Succeeding at Scale: Automated Multi-Retriever Fusion and Query-Side Adaptation for Multi-Tenant Search

TL;DR

The paper tackles the lack of relevance labels and the prohibitively costly re-indexing in large-scale multi-tenant retrieval. It introduces DevRev Search, a fully automated dataset pipeline that fuses candidates from multiple retrievers and uses LLM-based filtering, paired with an index-preserving adaptation that fine-tunes only the query encoder via LoRA. The work demonstrates that targeting specific query-layer configurations yields favorable quality-efficiency trade-offs, with competitive performance on DevRev Search and SciFact while avoiding re-indexing overhead. This approach offers a scalable path for personalized enterprise search across thousands of tenants by enabling rapid model updates without rebuilding the document index.

Abstract

Large-scale multi-tenant retrieval systems amass vast user query logs yet critically lack the curated relevance labels required for effective domain adaptation. This "dark data" problem is exacerbated by the operational cost of model updates: jointly fine-tuning query and document encoders requires re-indexing the entire corpus, which is prohibitive in multi-tenant environments with thousands of isolated indices. To address these dual challenges, we introduce \textbf{DevRev Search}, a passage retrieval benchmark for technical customer support constructed through a fully automatic pipeline. We employ a \textbf{fusion-based candidate generation} strategy, pooling results from diverse sparse and dense retrievers, and utilize an LLM-as-a-Judge to perform rigorous \textbf{consistency filtering} and relevance assignment. We further propose a practical \textbf{Index-Preserving Adaptation} strategy: by fine-tuning only the query encoder via Low-Rank Adaptation (LoRA), we achieve competitive performance improvements while keeping the document index frozen. Our experiments on DevRev Search and SciFact demonstrate that targeting specific transformer layers in the query encoder yields optimal quality-efficiency trade-offs, offering a scalable path for personalized enterprise search.

Figures (3)

• Figure 1: Comparison of Recall@10 for Baseline, Query-Document ($QD$), and Query-Only ($Q$) fine-tuning. On DevRev Search, $Q$ surprisingly outperforms $QD$, while on SciFact, $Q$ remains highly competitive.
• Figure 2: Performance vs. LoRA Rank. Note Qwen's preference for lower ranks ($r=8$) on DevRev Search versus mid-ranks ($r=64$) on SciFact.
• Figure 3: Targeted Module Fine Tuning