M3Retrieve: Benchmarking Multimodal Retrieval for Medicine

TL;DR

M3Retrieve introduces the first large-scale, multimodal medical retrieval benchmark designed to evaluate text-and-image guided information access across 16 medical disciplines and five retrieval tasks. It integrates data from diverse open sources and employs clinician-informed relevance mappings to create challenging ground-truth for VisualContext, Multimodal Summary, Multimodal Query-to-Image, and Case Study Retrieval, plus quality control via domain experts. The study benchmarks a spectrum of retrieval models—from traditional BM25 to dense uni-modal and multimodal encoders and late-interaction architectures—revealing that multimodal models excel when cross-modal signals are essential, while text-focused encoders currently dominate purely textual tasks. By providing a publicly available dataset and baselines, M3Retrieve offers a principled platform for evaluating medical multimodal retrieval systems and catalyzing the development of reliable AI tools for clinical decision support.

Abstract

With the increasing use of RetrievalAugmented Generation (RAG), strong retrieval models have become more important than ever. In healthcare, multimodal retrieval models that combine information from both text and images offer major advantages for many downstream tasks such as question answering, cross-modal retrieval, and multimodal summarization, since medical data often includes both formats. However, there is currently no standard benchmark to evaluate how well these models perform in medical settings. To address this gap, we introduce M3Retrieve, a Multimodal Medical Retrieval Benchmark. M3Retrieve, spans 5 domains,16 medical fields, and 4 distinct tasks, with over 1.2 Million text documents and 164K multimodal queries, all collected under approved licenses. We evaluate leading multimodal retrieval models on this benchmark to explore the challenges specific to different medical specialities and to understand their impact on retrieval performance. By releasing M3Retrieve, we aim to enable systematic evaluation, foster model innovation, and accelerate research toward building more capable and reliable multimodal retrieval systems for medical applications. The dataset and the baselines code are available in this github page https://github.com/AkashGhosh/M3Retrieve.

Figures (3)

• Figure 1: M3Retreive is a multimodal medical retrieval benchmark comprising samples from four different tasks across multiple healthcare subdomains obtained from a variety of open-sourced data sources resulting in total dataset size of about 800K query-corpus pairs. It encompasses testing of retrieval models across varied retrieval paradigms
• Figure 2: Overview of a retrieval task addressed in the M3Retrieve Benchmark. The task aims to integrate both text and image data, with the retriever model ranking documents based on relevance. The multimodal framework enriches retrieval performance by incorporating visual information alongside traditional text-based retrieval.
• Figure 3: An example from the M3Retrieve Benchmark showing the query image-text pair and corpus texts along with justifications for the assigned scores.