Typologically Informed Parameter Aggregation

TL;DR

TIPA addresses cross-lingual transfer for underrepresented languages by constructing proxy adapters through typology-guided aggregation of existing adapters, all without training. A proxy adapter $L_{ ext{proxy}}$ is formed by layer-wise parameter aggregation weighted by $w_s = rac{ ext{exp}(1 - d( ext{l}_{ ext{tgt}}, ext{l}_{ ext{src}}))}{ ext{\sum}_{ ext{l}_{ ext{src}}'} ext{exp}(1 - d( ext{l}_{ ext{tgt}}, ext{l}_{ ext{src}}'))}$, where distances $d(\cdot, ext{\cdot})$ come from URIEL+ typological data. The proxy adapters integrate into the MAD-X framework to enable zero-shot transfer with a fixed task adapter. Evaluations across five NLP tasks and 234 languages show that TIPA consistently matches or beats strong baselines, with the largest benefits for languages without dedicated adapters and without any language-specific training.

Abstract

Massively multilingual language models enable cross-lingual generalization but underperform on low-resource and unseen languages. While adapter-based fine-tuning offers a parameter-efficient solution, training language-specific adapters at scale remains costly. We introduce Typologically Informed Parameter Aggregation (TIPA), a training-free method that constructs proxy language adapters by aggregating existing ones, weighted by typological similarity. Integrated into the MAD-X framework, these proxies enable zero-shot cross-lingual transfer without additional training. We evaluate TIPA on five NLP tasks and over 230 languages. TIPA consistently outperforms or matches baselines such as English-only fine-tuning or selecting the typologically closest language adapter. We see the largest gains for languages lacking dedicated adapters. Our results demonstrate that typologically informed aggregation provides a viable alternative to language-specific modules without any training needed.

Figures (4)

• Figure 1: Typologically Informed Parameter Aggregation (TIPA): a training-free framework that constructs a proxy language adapter for a target language by aggregating existing ones, weighted by typological similarity.
• Figure 2: Scores for all tasks across available languages. We compare TIPA to No Train but Gain, fine-tuning, uniform parameter averaging, and to either MAD-X or the typologically closest adapter, depending on adapter availability. Languages are marked for inclusion in XLM-R, and ordered based on increasing performance of our method. Larger formats of each presented figure can be found in §\ref{['sec:task_level_results']}, alongside a more detailed analysis.
• Figure 3: Scores for each task, across all evaluated languages. Comparing TIPA to the No Train but Gain and fine-tuning baselines, as well as to either the MAD-X configuration or the typologically closest adapter, depending on adapter availability. Languages are presented in order of increasing performance of our method, and marked for native support in XLM-R. The results show that languages not included in XLM-R pre-training generally underperform, and that the benefits of our method primarily concern natively supported languages.
• Figure 4: Scores for each task, across all evaluated languages. Comparing TIPA to the No Train but Gain and fine-tuning baselines, as well as to either the MAD-X configuration or the typologically closest adapter, depending on adapter availability. Languages are presented in order of increasing performance of our method, and marked for native support in XLM-R. The results show that languages not included in XLM-R pre-training generally underperform, and that the benefits of our method primarily concern natively supported languages.