Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain

TL;DR

Medical mT5 introduces the first open-source multilingual text-to-text LLM for the medical domain, trained on the largest public multilingual medical corpus to date across English, Spanish, French, and Italian. By continuing pre-training a multilingual T5 (mT5) model on domain data and releasing two model sizes (770M and 3B parameters), the work delivers strong multilingual performance, especially in multi-task and zero-shot cross-lingual settings, while maintaining practical hardware requirements. The authors also generate two new multilingual benchmarks for argument mining and abstractive QA to catalyze cross-language research. Overall, Medical mT5 demonstrates competitive English performance and clear advantages in non-English languages, enabling accessible, multilingual medical NLP research and applications with open resources.

Abstract

Research on language technology for the development of medical applications is currently a hot topic in Natural Language Understanding and Generation. Thus, a number of large language models (LLMs) have recently been adapted to the medical domain, so that they can be used as a tool for mediating in human-AI interaction. While these LLMs display competitive performance on automated medical texts benchmarks, they have been pre-trained and evaluated with a focus on a single language (English mostly). This is particularly true of text-to-text models, which typically require large amounts of domain-specific pre-training data, often not easily accessible for many languages. In this paper, we address these shortcomings by compiling, to the best of our knowledge, the largest multilingual corpus for the medical domain in four languages, namely English, French, Italian and Spanish. This new corpus has been used to train Medical mT5, the first open-source text-to-text multilingual model for the medical domain. Additionally, we present two new evaluation benchmarks for all four languages with the aim of facilitating multilingual research in this domain. A comprehensive evaluation shows that Medical mT5 outperforms both encoders and similarly sized text-to-text models for the Spanish, French, and Italian benchmarks, while being competitive with current state-of-the-art LLMs in English.

Figures (2)

• Figure 1: Text-to-Text representation of the Sequence Labeling task. Given an input sentence, the model is expected to generate the same sentence annotated with html-style tags.
• Figure 2: Text-to-Text representation of the BioASQ task. Given a question and a set of relevant snippets, the model generates an answer.