Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT
Jenny Kunz
TL;DR
The study addresses adapting a small Icelandic LLM to native language data using unstructured text with parameter-efficient fine-tuning (PEFT) while preserving long-context capabilities. It systematically compares PEFT methods and placements, finding that increasing trainable parameters helps and that LoRA in feed-forward layers yields the best results, with bottleneck adapters as a strong alternative. Prefix tuning and IA$^3$ underperform for language adaptation, and attention-based LoRA requires more parameters to match feed-forward LoRA. Data quality matters: CCNet-curated Icelandic data generally outperforms IGC for summarisation tasks, and extending context length may require last-layer adaptation or careful layer-focused strategies; future work should explore larger models, more data, and human evaluations to better capture language-specific generation quality.
Abstract
Smaller LLMs still face significant challenges even in medium-resourced languages, particularly when it comes to language-specific knowledge -- a problem not easily resolved with machine-translated data. In this case study on Icelandic, we aim to enhance the generation performance of an LLM by specialising it using unstructured text corpora. A key focus is on preventing interference with the models' capabilities of handling longer context during this adaptation. Through ablation studies using various parameter-efficient fine-tuning (PEFT) methods and setups, we find that increasing the number of trainable parameters leads to better and more robust language adaptation. LoRAs placed in the feed-forward layers and bottleneck adapters show promising results with sufficient parameters, while prefix tuning and (IA)3 are not suitable. Although improvements are consistent in 0-shot summarisation, some adapted models struggle with longer context lengths, an issue that can be mitigated by adapting only the final layers.