Soft Prompt Tuning for Cross-Lingual Transfer: When Less is More

TL;DR

This paper addresses the challenge of cross-lingual transfer with large multilingual language models by applying Soft Prompt Tuning (SPT) while keeping all model parameters frozen, enabling substantial parameter and compute savings in few-shot settings. The authors study the impact of prompt length and prompt reparameterization across 52 languages using two model families, XGLM and BLOOM, on the SIB-200 dataset derived from FLORES-200. They find that freezing the model generally improves cross-lingual transfer, reduces bias toward linguistically similar languages, and that shorter prompts are typically more effective, while reparameterization yields language-dependent gains and losses, being particularly sensitive in BLOOM. These results demonstrate that parameter-efficient SPT scales to multilingual, large-scale models and offer practical guidance for deploying NLP in low-resource languages, with future work aiming to balance performance across diverse languages.

Abstract

Soft Prompt Tuning (SPT) is a parameter-efficient method for adapting pre-trained language models (PLMs) to specific tasks by inserting learnable embeddings, or soft prompts, at the input layer of the PLM, without modifying its parameters. This paper investigates the potential of SPT for cross-lingual transfer. Unlike previous studies on SPT for cross-lingual transfer that often fine-tune both the soft prompt and the model parameters, we adhere to the original intent of SPT by keeping the model parameters frozen and only training the soft prompt. This does not only reduce the computational cost and storage overhead of full-model fine-tuning, but we also demonstrate that this very parameter efficiency intrinsic to SPT can enhance cross-lingual transfer performance to linguistically distant languages. Moreover, we explore how different factors related to the prompt, such as the length or its reparameterization, affect cross-lingual transfer performance.

Figures (5)

• Figure 1: A simplified illustration of SPT lester_power_2021. $P_1, \ldots, P_n$ denote the soft prompt tokens, with each token corresponding to a trainable embedding. Essentially, for a model with an embedding dimension $d$, a soft prompt of length $n$ forms a $d \times n$ matrix.
• Figure 2: Average cross-lingual transfer performance of SPT with and without model freezing (MF) for different models across all languages supported by the respective model.
• Figure 3: Average cross-lingual transfer performance of SPT with model freezing for different number of training samples per class.
• Figure 4: Average cross-lingual transfer performance, measured as accuracy, across different prompt lengths for different models.
• Figure 5: Impact of reparameterization (expressed in %) on the cross-lingual transfer performance of BLOOM-560M and XGLM-564M for different target languages.