MedEIR: A Specialized Medical Embedding Model for Enhanced Information Retrieval

TL;DR

MedEIR tackles the challenge of medical retrieval with embeddings that perform well across both medical and general domains while handling long documents. It integrates a domain-adapted tokenizer, ALiBi long-context processing, and a three-stage training regimen (MLM pretraining on ~6B tokens, large-scale contrastive pretraining, and hard-negative fine-tuning) to produce high-quality embeddings with a 52,543-token vocabulary. Empirically, MedEIR achieves top results on multiple medical benchmarks (e.g., ArguAna 55.24, NFCorpus 38.44, MedicalQARetrieval 74.25, SciFact 72.04, TRECCOVID 79.56) and shows strong BEIR performance, while reducing token fragmentation by ~30% and memory usage by up to ~20%. The work demonstrates that combining domain-specific tokenization, scalable long-context encoding via ALiBi, and multi-stage contrastive fine-tuning yields robust, versatile embeddings suitable for Retrieval-Augmented Generation and related medical NLP tasks, with broad practical impact for scalable clinical information retrieval.

Abstract

Embedding models have become essential for retrieval-augmented generation (RAG) tasks, semantic clustering, and text re-ranking. But despite their growing use, many of these come with notable limitations. For example, Jina fails to capture the semantic content of medical documents, while models such as MiniLM often perform poorly on long-form documents. Domain-adapted models, while specialized, often underperform in general-purpose tasks, reducing their overall applicability. General-domain tokenizers often misinterpret medical vocabulary. The limitations of current embedding models, whether in tokenization accuracy, domain comprehension, or handling long sequences, highlight the need for more versatile solutions. In this work, we present MedEIR, a novel embedding model and tokenizer jointly optimized for both medical and general NLP tasks, incorporating ALiBi-based long-context processing to support sequences of up to 8,192 tokens. MedEIR was pre-trained on only 6 billion tokens, significantly fewer than Jina's, followed by fine-tuning on 3 million sentence pairs. MedEIR consistently outperforms Jina V2 and MiniLM across MTEB benchmarks, achieving top scores on ArguAna (55.24), NFCorpus (38.44), MedicalQARetrieval (74.25), SciFact (72.04), and TRECCOVID (79.56). These results highlight the potential of MedEIR as a highly effective embedding model, demonstrating strong performance across both general-purpose and domain-specific tasks and outperforming existing models on multiple benchmarks.

Figures (1)

• Figure 1: Comparison of Our Model on MTEB Medical Benchmark Datasets