CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Data

TL;DR

The work addresses the need for French biomedical NLP models trained on domain-specific data. It proposes CamemBERT-bio, obtained by continual pre-trainingCamemBERT on a public biomedical corpus (biomed-fr) and evaluates it across multiple French biomedical NER tasks using standardized protocols. The results show a consistent average improvement of $2.54$ points in F1 over CamemBERT, establishing continual pre-training as a cost-effective alternative to training from scratch, with ecological advantages. The study also highlights the importance of consistent evaluation benchmarks for fair comparisons and discusses tokenization, dataset diversity, and future directions for expanding public biomedical resources in French.

Abstract

Clinical data in hospitals are increasingly accessible for research through clinical data warehouses. However these documents are unstructured and it is therefore necessary to extract information from medical reports to conduct clinical studies. Transfer learning with BERT-like models such as CamemBERT has allowed major advances for French, especially for named entity recognition. However, these models are trained for plain language and are less efficient on biomedical data. Addressing this gap, we introduce CamemBERT-bio, a dedicated French biomedical model derived from a new public French biomedical dataset. Through continual pre-training of the original CamemBERT, CamemBERT-bio achieves an improvement of 2.54 points of F1-score on average across various biomedical named entity recognition tasks, reinforcing the potential of continual pre-training as an equally proficient yet less computationally intensive alternative to training from scratch. Additionally, we highlight the importance of using a standard evaluation protocol that provides a clear view of the current state-of-the-art for French biomedical models.