ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model

TL;DR

This work addresses the underrepresentation of Angolan languages in multilingual PLMs by developing four MAFT-based models tailored to Umbundu, Kimbundu, Kikongo, Chokwe, and Luba-Kasai. It leverages informed embedding initialization via OFA and synthetic data augmentation to boost downstream tasks, demonstrating substantial gains over baselines such as XLM-R and AfroXLMR. Key findings show that region-specific MAFT with synthetic data and OFA initialization yields the best performance, with AngOFA achieving notable improvements (e.g., +16.6 over XLM-R and +12.3 over AfroXLMR) on SIB-200 evaluated Angolan languages. The study suggests that combining region-focused pretraining with embedding-aware initialization and data augmentation can meaningfully close the resource gap for low-resource languages and motivates extending OFA to more African languages.

Abstract

In recent years, the development of pre-trained language models (PLMs) has gained momentum, showcasing their capacity to transcend linguistic barriers and facilitate knowledge transfer across diverse languages. However, this progress has predominantly bypassed the inclusion of very-low resource languages, creating a notable void in the multilingual landscape. This paper addresses this gap by introducing four tailored PLMs specifically finetuned for Angolan languages, employing a Multilingual Adaptive Fine-tuning (MAFT) approach. In this paper, we survey the role of informed embedding initialization and synthetic data in enhancing the performance of MAFT models in downstream tasks. We improve baseline over SOTA AfroXLMR-base (developed through MAFT) and OFA (an effective embedding initialization) by 12.3 and 3.8 points respectively.