Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Resources and Few-shot Learners for In-context Learning in Slavic Languages

Michal Štefánik, Marek Kadlčík, Piotr Gramacki, Petr Sojka

TL;DR

This work tackles the scarcity of in-context learning resources for non-English languages by building an evaluation suite for Czech, Polish, and Russian, transforming diverse task data (NER, sentiment, QA, NLI) into a unified instruction format with native-language templates. It benchmarks contemporary ICL approaches (e.g., Tk-Instruct, mTk-Instruct, FLAN-T5) and trains language-specific ICL models, examining multilingual fine-tuning, prompting strategies, and model scale. Key findings show that multilingual instruction tuning generally enhances ICL in Slavic languages, while gains are task- and language-dependent; single-task QA-focused ICL models can compete with or surpass multi-task ICL models and even approach supervised baselines. The study also demonstrates strong cross-lingual transfer when English QA data is integrated and publicly releases templates, transformed datasets, and the new language-specific ICL models to support broader adoption and future research.

Abstract

Despite the rapid recent progress in creating accurate and compact in-context learners, most recent work focuses on in-context learning (ICL) for tasks in English. However, the ability to interact with users of languages outside English presents a great potential for broadening the applicability of language technologies to non-English speakers. In this work, we collect the infrastructure necessary for training and evaluation of ICL in a selection of Slavic languages: Czech, Polish, and Russian. We link a diverse set of datasets and cast these into a unified instructional format through a set of transformations and newly-crafted templates written purely in target languages. Using the newly-curated dataset, we evaluate a set of the most recent in-context learners and compare their results to the supervised baselines. Finally, we train, evaluate and publish a set of in-context learning models that we train on the collected resources and compare their performance to previous work. We find that ICL models tuned in English are also able to learn some tasks from non-English contexts, but multilingual instruction fine-tuning consistently improves the ICL ability. We also find that the massive multitask training can be outperformed by single-task training in the target language, uncovering the potential for specializing in-context learners to the language(s) of their application.

Resources and Few-shot Learners for In-context Learning in Slavic Languages

TL;DR

This work tackles the scarcity of in-context learning resources for non-English languages by building an evaluation suite for Czech, Polish, and Russian, transforming diverse task data (NER, sentiment, QA, NLI) into a unified instruction format with native-language templates. It benchmarks contemporary ICL approaches (e.g., Tk-Instruct, mTk-Instruct, FLAN-T5) and trains language-specific ICL models, examining multilingual fine-tuning, prompting strategies, and model scale. Key findings show that multilingual instruction tuning generally enhances ICL in Slavic languages, while gains are task- and language-dependent; single-task QA-focused ICL models can compete with or surpass multi-task ICL models and even approach supervised baselines. The study also demonstrates strong cross-lingual transfer when English QA data is integrated and publicly releases templates, transformed datasets, and the new language-specific ICL models to support broader adoption and future research.

Abstract

Despite the rapid recent progress in creating accurate and compact in-context learners, most recent work focuses on in-context learning (ICL) for tasks in English. However, the ability to interact with users of languages outside English presents a great potential for broadening the applicability of language technologies to non-English speakers. In this work, we collect the infrastructure necessary for training and evaluation of ICL in a selection of Slavic languages: Czech, Polish, and Russian. We link a diverse set of datasets and cast these into a unified instructional format through a set of transformations and newly-crafted templates written purely in target languages. Using the newly-curated dataset, we evaluate a set of the most recent in-context learners and compare their results to the supervised baselines. Finally, we train, evaluate and publish a set of in-context learning models that we train on the collected resources and compare their performance to previous work. We find that ICL models tuned in English are also able to learn some tasks from non-English contexts, but multilingual instruction fine-tuning consistently improves the ICL ability. We also find that the massive multitask training can be outperformed by single-task training in the target language, uncovering the potential for specializing in-context learners to the language(s) of their application.
Paper Structure (27 sections, 1 equation, 1 figure, 6 tables)

This paper contains 27 sections, 1 equation, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. In-context learners
  4. In-context Few-shot learning
  5. Datasets
  6. Data Collections in Target Languages
  7. Czech Datasets
  8. Polish
  9. Russian
  10. Templates
  11. Experiments
  12. In-context Few-shot Learning Evaluation
  13. Multilingual fine-tuning
  14. Fine-tuning strategy
  15. Model size
  16. ...and 12 more sections

Figures (1)

  • Figure 1: In this work, we transform Czech, Polish, and Russian datasets for diverse task types into a unified instructional format through a set of templates curated by the native speakers of target languages. The resulting collection enables an evaluation of existing in-context learners as well as the creation of new in-context learners interacting fully in the target language.