Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CreoleVal: Multilingual Multitask Benchmarks for Creoles

Heather Lent, Kushal Tatariya, Raj Dabre, Yiyi Chen, Marcell Fekete, Esther Ploeger, Li Zhou, Ruth-Ann Armstrong, Abee Eijansantos, Catriona Malau, Hans Erik Heje, Ernests Lavrinovics, Diptesh Kanojia, Paul Belony, Marcel Bollmann, Loïc Grobol, Miryam de Lhoneux, Daniel Hershcovich, Michel DeGraff, Anders Søgaard, Johannes Bjerva

TL;DR

Creole languages have been largely missing from multilingual NLP benchmarks due to data scarcity and sociolinguistic stigma. CreoleVal introduces a broad, cross-task benchmark suite covering reading comprehension, relation classification, and machine translation across up to 28 Creoles, complemented by existing tasks, to enable systematic zero-shot transfer studies with popular multilingual LMs. The work demonstrates both the promise and challenges of cross-lingual transfer in Creoles, showing how model choice, data domain, and resource quality shape performance in NLU and NLG tasks. By providing data, code, and baselines, CreoleVal aims to catalyze equitable NLP research and the development of language technologies for Creole-speaking communities, while also highlighting domain diversity and modality as critical future directions.

Abstract

Creoles represent an under-explored and marginalized group of languages, with few available resources for NLP research.While the genealogical ties between Creoles and a number of highly-resourced languages imply a significant potential for transfer learning, this potential is hampered due to this lack of annotated data. In this work we present CreoleVal, a collection of benchmark datasets spanning 8 different NLP tasks, covering up to 28 Creole languages; it is an aggregate of novel development datasets for reading comprehension, relation classification, and machine translation for Creoles, in addition to a practical gateway to a handful of preexisting benchmarks. For each benchmark, we conduct baseline experiments in a zero-shot setting in order to further ascertain the capabilities and limitations of transfer learning for Creoles. Ultimately, we see CreoleVal as an opportunity to empower research on Creoles in NLP and computational linguistics, and in general, a step towards more equitable language technology around the globe.

CreoleVal: Multilingual Multitask Benchmarks for Creoles

TL;DR

Creole languages have been largely missing from multilingual NLP benchmarks due to data scarcity and sociolinguistic stigma. CreoleVal introduces a broad, cross-task benchmark suite covering reading comprehension, relation classification, and machine translation across up to 28 Creoles, complemented by existing tasks, to enable systematic zero-shot transfer studies with popular multilingual LMs. The work demonstrates both the promise and challenges of cross-lingual transfer in Creoles, showing how model choice, data domain, and resource quality shape performance in NLU and NLG tasks. By providing data, code, and baselines, CreoleVal aims to catalyze equitable NLP research and the development of language technologies for Creole-speaking communities, while also highlighting domain diversity and modality as critical future directions.

Abstract

Creoles represent an under-explored and marginalized group of languages, with few available resources for NLP research.While the genealogical ties between Creoles and a number of highly-resourced languages imply a significant potential for transfer learning, this potential is hampered due to this lack of annotated data. In this work we present CreoleVal, a collection of benchmark datasets spanning 8 different NLP tasks, covering up to 28 Creole languages; it is an aggregate of novel development datasets for reading comprehension, relation classification, and machine translation for Creoles, in addition to a practical gateway to a handful of preexisting benchmarks. For each benchmark, we conduct baseline experiments in a zero-shot setting in order to further ascertain the capabilities and limitations of transfer learning for Creoles. Ultimately, we see CreoleVal as an opportunity to empower research on Creoles in NLP and computational linguistics, and in general, a step towards more equitable language technology around the globe.
Paper Structure (48 sections, 2 figures, 8 tables)

This paper contains 48 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Inclusion of Creoles
  3. Contributions
  4. Background
  5. Linguistic Context
  6. Previous Work
  7. Transfer Learning
  8. Multilingual Language Models
  9. Natural Language Understanding of Creoles
  10. Reading Comprehension
  11. Translation
  12. Results and Analysis
  13. Relation Classification
  14. Model and Training
  15. Results and Analysis
  16. ...and 33 more sections

Figures (2)

  • Figure 1: CreoleVal expands the availability of labeled data for Creoles around the globe. This chart shows the increased availability of datasets for concrete tasks, across Creoles from different regions. Before CreoleVal, only 11 Creoles had data for at least 1 pre-existing task, and now 28 Creoles have labeled data for at least 1 task and at most 6 tasks.
  • Figure 2: chrF (blue color) BLEU (red color) scores obtained using baseline models (scratch; square points) and fine-tuned models (mBART-50-MT-FT; circle points) on the Bible corpus for XX-En (left) and En-XX (right) language pairs, where XX represents Creole languages. The language pairs are ordered from left to right in increasing sizes of parallel corpora from 4,366 pairs to 583,746 pairs