Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translation

TL;DR

This study tackles domain-adaptive NMT for low-resource languages by comparing fine-tuning and intermediate-task transfer learning on multilingual language models, using auxiliary-domain parallel data. It systematically evaluates bitext denoising pre-training and various FT/ITTL configurations across in-domain and out-domain scenarios, quantified with spBLEU and domain-divergence metrics. The key finding is that pre-training with parallel data often does not outperform vanilla fine-tuning at small data sizes, while multi-domain ITTL can provide gains in-domain, especially with modest auxiliary data, though benefits diminish as target-domain data grows; out-domain results are heavily contingent on domain similarity, with divergence quantified via Jensen-Shannon Divergence. These insights guide practical strategies for leveraging auxiliary domains in domain-specific LRL-NMT and highlight the role of domain divergence in shaping transfer learning effectiveness.

Abstract

Neural Machine Translation (NMT) systems built on multilingual sequence-to-sequence Language Models (msLMs) fail to deliver expected results when the amount of parallel data for a language, as well as the language's representation in the model are limited. This restricts the capabilities of domain-specific NMT systems for low-resource languages (LRLs). As a solution, parallel data from auxiliary domains can be used either to fine-tune or to further pre-train the msLM. We present an evaluation of the effectiveness of these two techniques in the context of domain-specific LRL-NMT. We also explore the impact of domain divergence on NMT model performance. We recommend several strategies for utilizing auxiliary parallel data in building domain-specific NMT models for LRLs.

Figures (9)

• Figure 1: Fine-tuning Strategies. (a) Vanilla FT (b) Mixed-domain FT (c) Single-domain ITTL (d) Multi-domain ITTL. $D$- set of all domain-specific datasets per language pair. $d_i$, $d_j$, $d_k$ - domain-specific datasets.
• Figure 2: In-domain testing for different methods.
• Figure 3: Out-domain testing for different methods.
• Figure 4: Single-domain ITTL results on in-domain cases.
• Figure 5: Single-domain ITTL results for out-domain cases.