AutoMIR: Effective Zero-Shot Medical Information Retrieval without Relevance Labels

TL;DR

This paper introduces SL-HyDE, a fully zero-shot dense retrieval framework for medical information retrieval that leverages an LLM as a generator to produce hypothetical documents conditioned on queries and a retriever that ranks real documents using these pseudo-documents. A self-learning loop jointly optimizes both components using unlabeled medical corpora, obviating the need for relevance labels. To evaluate the approach in Chinese MIR, the authors propose CMIRB, a realistic benchmark spanning five tasks and ten datasets with careful data curation. Empirical results show SL-HyDE consistently improves over HyDE and various baselines across CMIRB, demonstrating strong generalization across generators and retrievers and highlighting the framework’s scalability and practical impact for low-resource medical retrieval. The work also establishes a first comprehensive Chinese MIR benchmark, motivating future research in domain-adaptive, zero-shot information retrieval in medicine and beyond.

Abstract

Medical information retrieval (MIR) is essential for retrieving relevant medical knowledge from diverse sources, including electronic health records, scientific literature, and medical databases. However, achieving effective zero-shot dense retrieval in the medical domain poses substantial challenges due to the lack of relevance-labeled data. In this paper, we introduce a novel approach called \textbf{S}elf-\textbf{L}earning \textbf{Hy}pothetical \textbf{D}ocument \textbf{E}mbeddings (\textbf{SL-HyDE}) to tackle this issue. SL-HyDE leverages large language models (LLMs) as generators to generate hypothetical documents based on a given query. These generated documents encapsulate key medical context, guiding a dense retriever in identifying the most relevant documents. The self-learning framework progressively refines both pseudo-document generation and retrieval, utilizing unlabeled medical corpora without requiring any relevance-labeled data. Additionally, we present the Chinese Medical Information Retrieval Benchmark (CMIRB), a comprehensive evaluation framework grounded in real-world medical scenarios, encompassing five tasks and ten datasets. By benchmarking ten models on CMIRB, we establish a rigorous standard for evaluating medical information retrieval systems. Experimental results demonstrate that SL-HyDE significantly surpasses HyDE in retrieval accuracy while showcasing strong generalization and scalability across various LLM and retriever configurations. Our code and data are publicly available at: https://github.com/ll0ruc/AutoMIR

Figures (6)

• Figure 1: Training and inference pipeline of SL-HyDE.
• Figure 2: An overview of CMIRB.
• Figure 3: CMIRB benchmark construction pipeline.
• Figure 4: Prompt for data processing (I).
• Figure 5: Prompt for data processing (II).