MEDBERT.de: A Comprehensive German BERT Model for the Medical Domain

TL;DR

MedBERT.de addresses the need for robust German clinical language understanding by pre-training a domain-specific BERT model on a large, diverse German medical corpus. The authors benchmark medBERT.de against general and other medical German models across radiology, open, and private medical datasets, showing state-of-the-art or competitive performance, particularly on longer texts such as radiology reports. Key contributions include a comprehensive benchmarking suite with radiology, private, and open data, an analysis of data deduplication and tokenizer fertility, and the public release of model weights and radiology benchmarks for reproducibility. The work highlights the primacy of data scale and domain alignment for German clinical NLP, while noting limitations related to dataset composition and institutional scope, and outlining future paths to broaden coverage across medical specialties.

Abstract

This paper presents medBERTde, a pre-trained German BERT model specifically designed for the German medical domain. The model has been trained on a large corpus of 4.7 Million German medical documents and has been shown to achieve new state-of-the-art performance on eight different medical benchmarks covering a wide range of disciplines and medical document types. In addition to evaluating the overall performance of the model, this paper also conducts a more in-depth analysis of its capabilities. We investigate the impact of data deduplication on the model's performance, as well as the potential benefits of using more efficient tokenization methods. Our results indicate that domain-specific models such as medBERTde are particularly useful for longer texts, and that deduplication of training data does not necessarily lead to improved performance. Furthermore, we found that efficient tokenization plays only a minor role in improving model performance, and attribute most of the improved performance to the large amount of training data. To encourage further research, the pre-trained model weights and new benchmarks based on radiological data are made publicly available for use by the scientific community.