Generalist embedding models are better at short-context clinical semantic search than specialized embedding models

TL;DR

The paper addresses robustness in short-context clinical semantic search by benchmarking 19 embedding methods (generalist versus domain-specific) on a newly created, reproducible ICD-10-CM reformulation dataset. Using a Euclidean retrieval setup, generalist, sentence-transformer embeddings consistently outperform clinical baselines, with ClinicalBERT as the strongest clinical performer but still trailing the top generalists by substantial margins. The findings suggest that broader training data and robust language understanding in generalist models yield better resilience to rephrasing in medical text. This work provides a practical, reproducible benchmark and guidance for deploying robust short-context clinical retrieval systems in real-world settings.

Abstract

The increasing use of tools and solutions based on Large Language Models (LLMs) for various tasks in the medical domain has become a prominent trend. Their use in this highly critical and sensitive domain has thus raised important questions about their robustness, especially in response to variations in input, and the reliability of the generated outputs. This study addresses these questions by constructing a textual dataset based on the ICD-10-CM code descriptions, widely used in US hospitals and containing many clinical terms, and their easily reproducible rephrasing. We then benchmarked existing embedding models, either generalist or specialized in the clinical domain, in a semantic search task where the goal was to correctly match the rephrased text to the original description. Our results showed that generalist models performed better than clinical models, suggesting that existing clinical specialized models are more sensitive to small changes in input that confuse them. The highlighted problem of specialized models may be due to the fact that they have not been trained on sufficient data, and in particular on datasets that are not diverse enough to have a reliable global language understanding, which is still necessary for accurate handling of medical documents.

Figures (2)

• Figure 1: Exact code retrieval performances depending on model scope and embedding size.
• Figure 2: Negative CER rate of incorrect retrieved code performances depending on model scope and embedding size.