Language Fusion for Parameter-Efficient Cross-lingual Transfer

TL;DR

Fusion forLanguage Representations (FLARE) in adapters is proposed, a novel method that enhances representation quality and downstream performance for languages other than English while maintaining parameter efficiency and improving transfer performance.

Abstract

Limited availability of multilingual text corpora for training language models often leads to poor performance on downstream tasks due to undertrained representation spaces for languages other than English. This 'under-representation' has motivated recent cross-lingual transfer methods to leverage the English representation space by e.g. mixing English and 'non-English' tokens at the input level or extending model parameters to accommodate new languages. However, these approaches often come at the cost of increased computational complexity. We propose Fusion forLanguage Representations (FLARE) in adapters, a novel method that enhances representation quality and downstream performance for languages other than English while maintaining parameter efficiency. FLARE integrates source and target language representations within low-rank (LoRA) adapters using lightweight linear transformations, maintaining parameter efficiency while improving transfer performance. A series of experiments across representative cross-lingual natural language understanding tasks, including natural language inference, question-answering and sentiment analysis, demonstrate FLARE's effectiveness. FLARE achieves performance improvements of 4.9% for Llama 3.1 and 2.2% for Gemma~2 compared to standard LoRA fine-tuning on question-answering tasks, as measured by the exact match metric.

Figures (7)

• Figure 1: Fusion of source and target representations in LoRA adapters inserted within the query and value matrices. The representations are fused in the adapter bottlenecks and the outputs are added to the query and value outputs before softmax $\otimes$ activation.
• Figure 2: During the forward pass with FLARE, source language representations $x^{S}$ are processed by transformer block $i$ and before fusion with target language representations $x^{T}$. Source representations are obtained by inferencing the mPLM without the fusion adapters.
• Figure 3: Illustration of the FLARE MT variant where projected encoder representations from an MT model are directly fused with target language representations within the fusion adapters in the mPLM. Encoder representations from the MT model serve as latent translations, avoiding discretization in the decoder.
• Figure 4: Average performance differences on NusaX and TyDiQA for XLM-R Large using FLARE with MT models of different size.
• Figure 5: Average activation values for English and Acehnese in the first bottleneck query layer in XLM-R Large for the NusaX test set; add+relu fusion.