Decomposed Prompting: Probing Multilingual Linguistic Structure Knowledge in Large Language Models

TL;DR

This work tackles probing multilingual linguistic structure knowledge in large language models by reframing sequence labeling as a set of per-token prompts. It introduces decomposed prompting, which generates individual prompts for each token and optionally leverages few-shot demonstrations, to improve both accuracy and inference efficiency over iterative prompting, evaluated on UDPOS across 38 languages with English-centric and multilingual LLMs. The results show that decomposed prompting yields superior performance in zero- and few-shot settings, with notable speed-ups, and reveal insights into how English-centric models transfer linguistic knowledge across languages depending on proximity and script familiarity. The findings offer a practical, training-free method to assess multilingual linguistic competence in LLMs and lay groundwork for extending such probing to more complex linguistic tasks.

Abstract

Probing the multilingual knowledge of linguistic structure in LLMs, often characterized as sequence labeling, faces challenges with maintaining output templates in current text-to-text prompting strategies. To solve this, we introduce a decomposed prompting approach for sequence labeling tasks. Diverging from the single text-to-text prompt, our prompt method generates for each token of the input sentence an individual prompt which asks for its linguistic label. We test our method on the Universal Dependencies part-of-speech tagging dataset for 38 languages, using both English-centric and multilingual LLMs. Our findings show that decomposed prompting surpasses the iterative prompting baseline in efficacy and efficiency under zero- and few-shot settings. Moreover, our analysis of multilingual performance of English-centric LLMs yields insights into the transferability of linguistic knowledge via multilingual prompting.

Figures (9)

• Figure 1: Comparison of different prompting methods for sequence labeling.
• Figure 2: Analysis of decomposed promoting performance grouped by language family (a) and script type (b) under zero- and few-shot settings on Mistral. "IE" refers to the Indo-European language family. "L" (Low) represents languages that constitute less than 0.005% of the pretraining corpus, while "H" (High) denotes all other languages.
• Figure 3: Panorama of Mistral model's per-language performance. Each node symbolizes a distinct language. (a) shows the few-shot performance and (b) shows the difference between few- and zero-shot performance for each language.
• Figure 4: UD POS tag set.
• Figure 5: Distribution of languages by language family in the dataset.