DESIRE-ME: Domain-Enhanced Supervised Information REtrieval using Mixture-of-Experts
Pranav Kasela, Gabriella Pasi, Raffaele Perego, Nicola Tonellotto
TL;DR
Open-domain Q&A retrieval struggles with topic heterogeneity and limited cross-domain generalization. DESIRE-ME introduces a query-side Mixture-of-Experts (MoE) with a supervised gating mechanism trained on Wikipedia categories to assign per-domain weights and a set of domain-specific specializers to contextualize the query, combined via a pooling module. Across BEIR datasets (NaturalQuestions, HotpotQA, FEVER) and three dense retrievers (COCO-DR, COCO-DR XL, Contriever), DESIRE-ME yields consistent gains, including up to 12% in NDCG@10 and 22% in P@1, and demonstrates zero-shot transfer to Climate-FEVER when trained on FEVER. The work highlights the practicality of domain-aware MoE for robust open-domain IR, while noting limitations such as the need for domain labels and avenues for future work like soft labeling with LLMs and broader datasets.
Abstract
Open-domain question answering requires retrieval systems able to cope with the diverse and varied nature of questions, providing accurate answers across a broad spectrum of query types and topics. To deal with such topic heterogeneity through a unique model, we propose DESIRE-ME, a neural information retrieval model that leverages the Mixture-of-Experts framework to combine multiple specialized neural models. We rely on Wikipedia data to train an effective neural gating mechanism that classifies the incoming query and that weighs the predictions of the different domain-specific experts correspondingly. This allows DESIRE-ME to specialize adaptively in multiple domains. Through extensive experiments on publicly available datasets, we show that our proposal can effectively generalize domain-enhanced neural models. DESIRE-ME excels in handling open-domain questions adaptively, boosting by up to 12% in NDCG@10 and 22% in P@1, the underlying state-of-the-art dense retrieval model.