Polyglot Prompt: Multilingual Multitask PrompTraining

TL;DR

This work introduces Polyglot Prompting (PolyPrompt), a multilingual multitask prompting framework that places diverse tasks and languages into a single encoder–decoder model without task- or language-specific modules. Tasks are reformulated as sequence-to-sequence prompts and trained jointly on seven target datasets plus expanding high-resource datasets, enabling cross-task and cross-language transfer within a unified framework. The study provides an interpretable multilingual evaluation methodology and analyzes the impact of prompt design, language prompts, and dataset characteristics, revealing that cross-lingual prompts and unified prompts often yield the strongest gains, though results vary by language and dataset. Overall, the findings demonstrate the viability of monolithic multilingual multitask prompting and offer practical guidance on prompts and data expansion to maximize cross-lingual transfer.

Abstract

This paper aims for a potential architectural improvement for multilingual learning and asks: Can different tasks from different languages be modeled in a monolithic framework, i.e. without any task/language-specific module? The benefit of achieving this could open new doors for future multilingual research, including allowing systems trained on low resources to be further assisted by other languages as well as other tasks. We approach this goal by developing a learning framework named Polyglot Prompting to exploit prompting methods for learning a unified semantic space for different languages and tasks with multilingual prompt engineering. We performed a comprehensive evaluation of 6 tasks, namely topic classification, sentiment classification, named entity recognition, question answering, natural language inference, and summarization, covering 24 datasets and 49 languages. The experimental results demonstrated the efficacy of multilingual multitask prompt-based learning and led to inspiring observations. We also present an interpretable multilingual evaluation methodology and show how the proposed framework, multilingual multitask prompt training, works. We release all datasets prompted in the best setting and code.

Figures (8)

• Figure 1: Application of prompt technology in three different scenarios: Generic, Multitask (MT), Multilingual Multitask (MLMT). QA, Sum, and NLI represent different tasks, namely question answering, summarization, and natural language inference here. PLM represents pre-trained language model, and "zh", "en", "fr", "ja", "de","es" denote different languages.
• Figure 2: The proposed PolyPrompt framework for multilingual multitask prompt training.
• Figure 3: The relative performance improvement of PolyPrompt and its variants over the vanilla mT5 (mT5) at the language-level. "IE" denotes the "Indo-European". PolyP, PolyPE, PolyPEX, and PolyPEP are abbreviations for PolyPrompt, PolyPrompt+Expand, PolyPrompt+Expand+XLSum, and PolyPrompt+Expand+PANX.
• Figure 4: Dataset bias of PAWS-X and XNLI characterized by $\phi_{p}$ defined in Sec. \ref{['sec:exp2-approach']}.
• Figure 5: The exploration of the language and uniformity of prompt design. (a) is the performance gap between cross-lingual (CL) and in-lingual (IL) prompt templates, where PolyP, PolyPE, PolyPEX, and PolyPEP are abbreviations for PolyPrompt, PolyPrompt+Expand, PolyPrompt+Expand+XLSum, and PolyPrompt+Expand+PANX. (b) is the relative performance improvement of PolyPrompt with unified prompt templates versus diversified prompt templates (e.g. PolyP-v1). PolyP-v(x) ($x \in [1,5]$) represent $x$-th version of diversified prompt templates. The bluer color indicates that the model with the cross-lingual (unified) prompts outperforms the in-lingual (diversified) prompts more, while the redder color has the opposite meaning. The last column is the average relative improvement.