Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learning

TL;DR

This study evaluates English–Spanish clinical machine translation using three multilingual PLMs (Marian s-MPLM, WMT21fb xL-MPLM, NLLB) across ClinSpEn-2022 tasks to understand the trade-offs between model size, domain adaptation, and transfer learning. It finds that a smaller, carefully fine-tuned Marian model often outperforms larger models, and that transfer learning with WMT21fb can extend Spanish translation to clinical content not seen during pretraining. The authors emphasize that automatic MT metrics can mislead and that expert human evaluation (HOPE) is essential to gauge true translation quality in healthcare. The work demonstrates the potential of domain-focused data cleaning and cross-language transfer to advance clinical knowledge transformation, and provides openly accessible data for future research.

Abstract

We conduct investigations on clinical text machine translation by examining multilingual neural network models using deep learning such as Transformer based structures. Furthermore, to address the language resource imbalance issue, we also carry out experiments using a transfer learning methodology based on massive multilingual pre-trained language models (MMPLMs). The experimental results on three subtasks including 1) clinical case (CC), 2) clinical terminology (CT), and 3) ontological concept (OC) show that our models achieved top-level performances in the ClinSpEn-2022 shared task on English-Spanish clinical domain data. Furthermore, our expert-based human evaluations demonstrate that the small-sized pre-trained language model (PLM) won over the other two extra-large language models by a large margin, in the clinical domain fine-tuning, which finding was never reported in the field. Finally, the transfer learning method works well in our experimental setting using the WMT21fb model to accommodate a new language space Spanish that was not seen at the pre-training stage within WMT21fb itself, which deserves more exploitation for clinical knowledge transformation, e.g. to investigate into more languages. These research findings can shed some light on domain-specific machine translation development, especially in clinical and healthcare fields. Further research projects can be carried out based on our work to improve healthcare text analytics and knowledge transformation. Our data will be openly available for research purposes at https://github.com/HECTA-UoM/ClinicalNMT

Figures (13)

• Figure 1: Illustration of the Investigation Workflow
• Figure 2: Marian Pre-Trained NMT - Training Pipeline.
• Figure 3: Several Attention-based Transformer NMT structure google2017attention.
• Figure 4: Original Dense Transformer vs MoE Transformer costa2022no_NLLB
• Figure 5: MoE vs Conditional MoE costa2022no_NLLB