Family Matters: Language Transfer and Merging for Adapting Small LLMs to Faroese

TL;DR

This work investigates adapting small LLMs to Faroese by transferring from related Scandinavian languages, merging multiple transfer models, and comparing full fine-tuning with LoRA. It introduces two minimal-pair benchmarks, FoBLiMP and FoBCoMP, along with expert Faroese human evaluations to address data scarcity. Key findings show transfer is essential, with Icelandic often providing linguistic advantages and Danish aiding comprehension; merging can offer complementary strengths but yields mixed gains, and LoRA excels in linguistic acceptability for smaller models while full fine-tuning boosts comprehension and downstream performance. The research provides practical guidance for low-resource language adaptation and highlights the need for task-aware transfer and tuning decisions, backed by both automatic benchmarks and human judgments.

Abstract

We investigate how to adapt small, efficient LLMs to Faroese, a low-resource North Germanic language. Starting from English models, we continue pre-training on related Scandinavian languages, either individually or combined via merging, before fine-tuning on Faroese. We compare full fine-tuning with parameter-efficient tuning using LoRA, evaluating their impact on both linguistic accuracy and text comprehension. Due to the lack of existing Faroese evaluation data, we construct two new minimal-pair benchmarks from adapted and newly collected datasets and complement them with human evaluations by Faroese linguists. Our results demonstrate that transfer from related languages is crucial, though the optimal source language depends on the task: Icelandic enhances linguistic accuracy, whereas Danish boosts comprehension. Similarly, the choice between full fine-tuning and LoRA is task-dependent: LoRA improves linguistic acceptability and slightly increases human evaluation scores on the base model, while full fine-tuning yields stronger comprehension performance and better preserves model capabilities during downstream fine-tuning.

Figures (7)

• Figure 1: Win rates of Icelandic over English only. FoBLiMP (left), FoBCoMP (right).
• Figure 2: Win rates of all transfer setups over English only (i.e., any Scandinavian language is better than none. FoBLiMP (left), FoBCoMP (right).
• Figure 3: Win rates of transfer languages across models and adaptation setups. In case of a tie, we count both. FoBLiMP (left), FoBCoMP (right).
• Figure 4: Wins of LoRA vs. full FT for model twins on FoBLiMP. No merges (left), merges (right).
• Figure 5: Wins of LoRA vs. full FT for model twins in FoBCoMP. No merges (left), merges (right).