Cross-Prompt Encoder for Low-Performing Languages

TL;DR

The paper tackles cross-lingual transfer to low-performing languages in multilingual NLP by introducing a parameter-efficient Cross-Prompt Encoder (XPE) and a complementary Dual Soft Prompt (DUAL). XPE learns abstract, transferable patterns from multi-source typologically diverse languages, while DUAL blends these with language-specific soft prompts, achieving strong zero-shot and fully supervised results on the SIB-200 benchmark using an XLM-R large backbone with under $0.3\%$ trainable parameters. Across extensive experiments, XPE excels for low-performing languages and unseen targets, whereas DUAL provides broad adaptability, collectively surpassing several baselines including some zero-shot prompts and full-model fine-tuning in multiple configurations. The work demonstrates the value of multilingual supervision combined with prompt modularity for efficient cross-lingual transfer and offers publicly available code and pretrained prompts to support reproducibility and extension to other tasks and architectures.

Abstract

Soft prompts have emerged as a powerful alternative to adapters in parameter-efficient fine-tuning (PEFT), enabling large language models (LLMs) to adapt to downstream tasks without architectural changes or parameter updates. While prior work has focused on stabilizing training via parameter interaction in small neural prompt encoders, their broader potential for transfer across languages remains unexplored. In this paper, we demonstrate that a prompt encoder can play a central role in improving performance on low-performing languages - those that achieve poor accuracy even under full-model fine-tuning. We investigate a lightweight encoder paired with multi-source training on typologically diverse languages. We call this architecture-training combination the Cross-Prompt Encoder (XPE), and show that it advances the capture of abstract, transferable patterns across languages. To complement XPE, we propose a Dual Soft Prompt mechanism that combines an encoder-based prompt with a directly trained standard soft prompt. This hybrid design proves especially effective for target languages that benefit from both broadly shared structure and language-specific alignment. Text classification experiments with a transformer encoder (XLM-R) on the SIB-200 benchmark reveal a consistent trade-off: XPE is most effective for low-performing languages, while hybrid variants offer broader adaptability across multilingual settings.

Figures (2)

• Figure 1: Architectural setup of the three methodologies during training: (a) SPT – Standard Soft Prompt, (b) XPE – Cross-Prompt Encoder, and (c) DUAL – Dual Soft Prompt, a hybrid combining both prior approaches. Fire and snowflake icons indicate trainable and frozen parameters, respectively.
• Figure 2: Comparison of different methods on the SIB-200 dataset. We group languages by whether they are seen in the pre-training corpus of XLM-R. Source language groups are provided in parentheses alongside the methods, where J5 refers to the Joshi5 group. Languages are ordered by $\text{DUAL}^{\text{XPE-70}}$ (J5) performance in each group. All the methods are ZS-XLT, except for GPT-4, which is ZS prompting. It should be noted that RoSPrompts used English DBPedia14 as a source dataset (a topic detection task with a different label space) and also employed language-specific verbalizers, introducing further evaluation mismatches.