Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Performance Variations in Finetuned Translators of Ultra-Low Resource Languages: Do Linguistic Differences Matter?

Isabel Gonçalves, Paulo Cavalin, Claudio Pinhanez

TL;DR

This study investigates why finetuned MT for ultra-low-resource Indigenous languages yields divergent results, focusing on Guarani Mbya and Nheengatu translations to and from Portuguese. Using cleaned datasets and No Language Left Behind variants, the authors compare zeroshot and finetuned performance across model sizes and data sizes, revealing that linguistic differences likely drive remaining variation more than preprocessing choices or scale. Zero-shot performance is inadequate, and model size or dataset size explain only a portion of the gap, with Nheengatu benefiting far more from finetuning. The findings highlight the need for language-aware translation methods that account for typology (e.g., SVO/SOV balance) and morphological complexity, and they motivate expanding analyses to additional languages and alternative training strategies.

Abstract

Finetuning pre-trained language models with small amounts of data is a commonly-used method to create translators for ultra-low resource languages such as endangered Indigenous languages. However, previous works have reported substantially different performances with translators created using similar methodology and data. In this work we systematically explored possible causes of the performance difference, aiming to determine whether it was a product of different cleaning procedures, limitations of the pre-trained models, the size of the base model, or the size of the training dataset, studying both directions of translation. Our studies, using two Brazilian Indigenous languages, related but with significant structural linguistic characteristics, indicated none or very limited influence from those training factors, suggesting differences between languages may play a significant role in the ability to produce translators by fine-tuning pre-trained models.

Exploring Performance Variations in Finetuned Translators of Ultra-Low Resource Languages: Do Linguistic Differences Matter?

TL;DR

This study investigates why finetuned MT for ultra-low-resource Indigenous languages yields divergent results, focusing on Guarani Mbya and Nheengatu translations to and from Portuguese. Using cleaned datasets and No Language Left Behind variants, the authors compare zeroshot and finetuned performance across model sizes and data sizes, revealing that linguistic differences likely drive remaining variation more than preprocessing choices or scale. Zero-shot performance is inadequate, and model size or dataset size explain only a portion of the gap, with Nheengatu benefiting far more from finetuning. The findings highlight the need for language-aware translation methods that account for typology (e.g., SVO/SOV balance) and morphological complexity, and they motivate expanding analyses to additional languages and alternative training strategies.

Abstract

Finetuning pre-trained language models with small amounts of data is a commonly-used method to create translators for ultra-low resource languages such as endangered Indigenous languages. However, previous works have reported substantially different performances with translators created using similar methodology and data. In this work we systematically explored possible causes of the performance difference, aiming to determine whether it was a product of different cleaning procedures, limitations of the pre-trained models, the size of the base model, or the size of the training dataset, studying both directions of translation. Our studies, using two Brazilian Indigenous languages, related but with significant structural linguistic characteristics, indicated none or very limited influence from those training factors, suggesting differences between languages may play a significant role in the ability to produce translators by fine-tuning pre-trained models.

Paper Structure

This paper contains 18 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Guarani Mbya and Nheengatu
  4. Datasets, Models, and Metrics
  5. The Guarani Mbya Dataset
  6. The Nheengatu Dataset
  7. The Pre-Trained NLLB Models
  8. Evaluation Metric
  9. Experiments and Results
  10. Zeroshot Evaluation
  11. Basic Finetuning Evaluation
  12. Model Size Impact Evaluation
  13. Training Dataset Size Impact Evaluation
  14. Discussion
  15. Final discussion
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Distribution of the evaluation of the Zeroshot NLLB-600M model for Guarani Mbya and Nheengatu.
  • Figure 2: Distribution of evaluation of the finetuned NLLB 600M models for Guarani Mbya and Nheengatu.
  • Figure 3: Evaluation of different sizes of NLBB model (600M and 3B parameters) for Guarani Mbya.
  • Figure 4: Evaluation of different sizes of NLBB model (600M and 3B parameters) for Nheengatu.
  • Figure 5: Comparison of downsampling strategies for the Nheengatu training dataset and the Guarani Mbya baseline.