Pretraining Strategies using Monolingual and Parallel Data for Low-Resource Machine Translation
Idriss Nguepi Nguefack, Mara Finkelstein, Toadoum Sari Sakayo
TL;DR
This work tackles the challenge of machine translation for low-resource languages, with a focus on Lingala. It systematically compares pretraining strategies that combine monolingual and parallel data across multiple languages using two architectures, mBART and AfroBART. Across four pretraining regimes, results show that multi-language pretraining and leveraging both data types significantly improve Lingala translation quality, though gains depend on data scale and model architecture. The study underscores the value of inclusive, multilingual pretraining for underrepresented languages and provides public data/code to foster further research, while acknowledging data accessibility and evaluation limitations. Overall, the findings suggest that carefully designed multilingual pretraining can help bridge the MT performance gap for marginalized language communities.
Abstract
This research article examines the effectiveness of various pretraining strategies for developing machine translation models tailored to low-resource languages. Although this work considers several low-resource languages, including Afrikaans, Swahili, and Zulu, the translation model is specifically developed for Lingala, an under-resourced African language, building upon the pretraining approach introduced by Reid and Artetxe (2021), originally designed for high-resource languages. Through a series of comprehensive experiments, we explore different pretraining methodologies, including the integration of multiple languages and the use of both monolingual and parallel data during the pretraining phase. Our findings indicate that pretraining on multiple languages and leveraging both monolingual and parallel data significantly enhance translation quality. This study offers valuable insights into effective pretraining strategies for low-resource machine translation, helping to bridge the performance gap between high-resource and low-resource languages. The results contribute to the broader goal of developing more inclusive and accurate NLP models for marginalized communities and underrepresented populations. The code and datasets used in this study are publicly available to facilitate further research and ensure reproducibility, with the exception of certain data that may no longer be accessible due to changes in public availability.