Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SemiAdapt and SemiLoRA: Efficient Domain Adaptation for Transformer-based Low-Resource Language Translation with a Case Study on Irish

Josh McGiff, Nikola S. Nikolov

TL;DR

This work targets the high computational cost of transferring large multilingual models to low-resource language translation, specifically English→Irish. It introduces two semi-supervised, inference-efficient domain adaptation methods, SemiAdapt and SemiLoRA, and demonstrates that SemiAdapt can outperform full-domain fine-tuning while SemiLoRA can match or surpass full-model fine-tuning on several domains. Through domain-by-domain and dataset-level analyses, the study reveals the benefits of embedding-informed domain labeling and the strong potential of parameter-efficient tuning to close performance gaps with full fine-tuning. The authors release open-source Irish translation models and resources, offering practical avenues for researchers to develop robust domain-adaptive NMT for low-resource languages with constrained compute.

Abstract

Fine-tuning is widely used to tailor large language models for specific tasks such as neural machine translation (NMT). However, leveraging transfer learning is computationally expensive when fine-tuning large multilingual models with billions of parameters, thus creating a barrier to entry for researchers working on low-resource domains such as Irish translation. Parameter-efficient fine-tuning (PEFT) bridges this gap by training on a fraction of the original model parameters, with the Low-Rank Adaptation (LoRA) approach introducing small, trainable adapter layers. We introduce SemiAdapt and SemiLoRA as semi-supervised inference-efficient approaches that strengthen domain adaptation and lead to improved overall performance in NMT. We demonstrate that SemiAdapt can outperform full-domain fine-tuning, while most notably, SemiLoRA can propel PEFT methods to match or even outperform full-model fine-tuning. We further evaluate domain-by-dataset fine-tuning and demonstrate that our embedding-based inference methods perform especially well on larger and noisier corpora. All Irish translation models developed in this work are released as open resources. These methods aim to make high-quality domain adaptation and fine-tuning more accessible to researchers working with low-resource languages.

SemiAdapt and SemiLoRA: Efficient Domain Adaptation for Transformer-based Low-Resource Language Translation with a Case Study on Irish

TL;DR

This work targets the high computational cost of transferring large multilingual models to low-resource language translation, specifically English→Irish. It introduces two semi-supervised, inference-efficient domain adaptation methods, SemiAdapt and SemiLoRA, and demonstrates that SemiAdapt can outperform full-domain fine-tuning while SemiLoRA can match or surpass full-model fine-tuning on several domains. Through domain-by-domain and dataset-level analyses, the study reveals the benefits of embedding-informed domain labeling and the strong potential of parameter-efficient tuning to close performance gaps with full fine-tuning. The authors release open-source Irish translation models and resources, offering practical avenues for researchers to develop robust domain-adaptive NMT for low-resource languages with constrained compute.

Abstract

Fine-tuning is widely used to tailor large language models for specific tasks such as neural machine translation (NMT). However, leveraging transfer learning is computationally expensive when fine-tuning large multilingual models with billions of parameters, thus creating a barrier to entry for researchers working on low-resource domains such as Irish translation. Parameter-efficient fine-tuning (PEFT) bridges this gap by training on a fraction of the original model parameters, with the Low-Rank Adaptation (LoRA) approach introducing small, trainable adapter layers. We introduce SemiAdapt and SemiLoRA as semi-supervised inference-efficient approaches that strengthen domain adaptation and lead to improved overall performance in NMT. We demonstrate that SemiAdapt can outperform full-domain fine-tuning, while most notably, SemiLoRA can propel PEFT methods to match or even outperform full-model fine-tuning. We further evaluate domain-by-dataset fine-tuning and demonstrate that our embedding-based inference methods perform especially well on larger and noisier corpora. All Irish translation models developed in this work are released as open resources. These methods aim to make high-quality domain adaptation and fine-tuning more accessible to researchers working with low-resource languages.
Paper Structure (18 sections, 5 figures, 2 tables)

This paper contains 18 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multilingual Models
  4. Neural Machine Translation
  5. Low-Resource Challenges
  6. Experimental Setup
  7. Data
  8. Base Model
  9. Experiments
  10. SemiLoRA: Semi-Supervised LoRA Fine-Tuning
  11. SemiAdapt: Semi-Supervised Full-Model Fine-Tuning
  12. Domain-by-Dataset Fine-Tuning
  13. Full-Dataset Fine-Tuning (Baseline)
  14. Conclusion
  15. Limitations
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Comparison of BLEU scores by domain for LoRA and SemiLoRA models trained on three and four domains.
  • Figure 2: BLEU scores by domain comparing full-model fine-tuning and SemiAdapt models trained on three and four domains.
  • Figure 3: BLEU scores by domain comparing full-model fine-tuning and LoRA-based models trained on four domains.
  • Figure 4: BLEU scores by domain comparing full-model fine-tuning and LoRA-based models on the domain-by-dataset split.
  • Figure 5: BLEU scores by domain comparing full-dataset fine-tuning, full-model domain fine-tuning, and LoRA-based models on the four-domain split.