Integrating Pre-trained Language Model into Neural Machine Translation

TL;DR

This work addresses the challenge of leveraging large pretrained language models (PLMs) within neural machine translation (NMT) by introducing PiNMT, a modular framework that fuses PLM representations with NMT through a PLM Multi Layer Converter, Embedding Fusion, and Cosine Alignment. It also proposes training strategies—Separate Learning Rates and Dual Step Training—to manage incompatibility and knowledge transfer. The approach achieves state-of-the-art BLEU on IWSLT'14 En$ ightarrow$De, outperforming prior methods by up to 5.16 BLEU over a Transformer baseline and 1.55 BLEU over the previous best, demonstrating robust improvements across PMLC variants and embedding fusion schemes. The results highlight the practical potential of PLM integration for low-resource translation, while outlining design considerations for dimensionality, interaction mechanisms, and training regimes that can guide future work in multilingual PLM–NMT fusion.

Abstract

Neural Machine Translation (NMT) has become a significant technology in natural language processing through extensive research and development. However, the deficiency of high-quality bilingual language pair data still poses a major challenge to improving NMT performance. Recent studies have been exploring the use of contextual information from pre-trained language model (PLM) to address this problem. Yet, the issue of incompatibility between PLM and NMT model remains unresolved. This study proposes PLM-integrated NMT (PiNMT) model to overcome the identified problems. PiNMT model consists of three critical components, PLM Multi Layer Converter, Embedding Fusion, and Cosine Alignment, each playing a vital role in providing effective PLM information to NMT. Furthermore, two training strategies, Separate Learning Rates and Dual Step Training, are also introduced in this paper. By implementing the proposed PiNMT model and training strategy, we achieve state-of-the-art performance on the IWSLT'14 En$\leftrightarrow$De dataset. This study's outcomes are noteworthy as they demonstrate a novel approach for efficiently integrating PLM with NMT to overcome incompatibility and enhance performance.

Figures (2)

• Figure 1: The architecture of the PiNMT model
• Figure 2: Converter