Meta-Learning for Low-Resource Neural Machine Translation

TL;DR

This work introduces MetaNMT, a model-agnostic meta-learning approach to low-resource neural machine translation that learns a transferable initialization across many high-resource language pairs. It integrates a universal lexical representation to bridge vocabulary gaps across languages, enabling effective meta-learning despite diverse input/output spaces. Evaluated on 5 target languages with 18 source languages, MetaNMT consistently outperforms multilingual transfer learning, especially in extremely data-scarce settings, and remains robust as training data decreases. The results demonstrate a principled pathway to rapid adaptation in low-resource MT and point to broad potential for incorporating additional monolingual and multilingual data within a unified meta-learning framework.

Abstract

In this paper, we propose to extend the recently introduced model-agnostic meta-learning algorithm (MAML) for low-resource neural machine translation (NMT). We frame low-resource translation as a meta-learning problem, and we learn to adapt to low-resource languages based on multilingual high-resource language tasks. We use the universal lexical representation~\citep{gu2018universal} to overcome the input-output mismatch across different languages. We evaluate the proposed meta-learning strategy using eighteen European languages (Bg, Cs, Da, De, El, Es, Et, Fr, Hu, It, Lt, Nl, Pl, Pt, Sk, Sl, Sv and Ru) as source tasks and five diverse languages (Ro, Lv, Fi, Tr and Ko) as target tasks. We show that the proposed approach significantly outperforms the multilingual, transfer learning based approach~\citep{zoph2016transfer} and enables us to train a competitive NMT system with only a fraction of training examples. For instance, the proposed approach can achieve as high as 22.04 BLEU on Romanian-English WMT'16 by seeing only 16,000 translated words (~600 parallel sentences).

Figures (5)

• Figure 1: The graphical illustration of the training process of the proposed MetaNMT. For each episode, one task (language pair) is sampled for meta-learning. The boxes and arrows in blue are mainly involved in language-specific learning (§\ref{['sec:lsl']}), and those in purple in meta-learning (§\ref{['sec:ml']}).
• Figure 2: An intuitive illustration in which we use solid lines to represent the learning of initialization, and dashed lines to show the path of fine-tuning.
• Figure 3: BLEU scores reported on test sets for {Ro, Lv, Fi, Tr} to En, where each model is first learned from 6 source tasks (Es, Fr, It, Pt, De, Ru) and then fine-tuned on randomly sampled training sets with around 16,000 English tokens per run. The error bars show the standard deviation calculated from 5 runs.
• Figure 4: BLEU Scores w.r.t. the size of the target task's training set.
• Figure 5: The learning curves of BLEU scores on the validation task (Ro-En).