Adapting BigScience Multilingual Model to Unseen Languages

TL;DR

Adapting a pre-existing 1.3B multilingual model to unseen languages is explored through three lightweight strategies (embedding-only, embedding-then-adapters, embedding-and-adapters) across different pretraining checkpoints. The study shows that jointly training embeddings and adapters, and especially updating positional embeddings, yields the strongest cross-language NLI performance for German and Korean, with low data budgets sufficing for low-resource scenarios. Pretraining steps influence zero-shot capabilities, but downstream finetuning can recover much of the task knowledge, highlighting the value of task-specific supervision. The work provides practical guidance on efficient language expansion for large multilingual LMs and identifies key components that matter for effective adaptation, including position encoding and adapter capacity.

Abstract

We benchmark different strategies of adding new languages (German and Korean) into the BigScience's pretrained multilingual language model with 1.3 billion parameters that currently supports 13 languages. We investigate the factors that affect the language adaptability of the model and the trade-offs between computational costs and expected performance.

Figures (2)

• Figure 1: Three different language adaptation strategies: (a) Embedding-only strategy: finetune the embedding layer of the pretrained model, which is tied with the language modeling head. (b) Embedding-then-Adapters strategy: first finetune the embedding layer, then train the invertible adapter and language adapters. (c) Embedding-and-Adapters strategy: finetune the embedding layer, invertible and language adapters at the same time. Grey blocks indicate frozen layers whereas white blocks indicate trainable layers.
• Figure 2: Graph of test accuracy for NLI against total adapter capacity (after applying reduction factor) using the Embedding-and-Adapters adaptation.