Taxi1500: A Multilingual Dataset for Text Classification in 1500 Languages

TL;DR

Taxi1500 tackles the scarcity of multilingual evaluation data by building a large-scale, Bible-based text classification dataset spanning 1504 languages. It annotates English verses via crowdsourcing and propagates labels to other languages through verse-aligned translations in the Parallel Bible Corpus, enabling broad cross-lingual evaluation. The authors benchmark four pretrained multilingual models, revealing how language coverage and script type influence zero-shot and in-language transfer, and they examine data-size effects and language-family dynamics. By releasing Taxi1500-c and the accompanying code, the work aims to accelerate research and evaluation for low-resource languages in multilingual NLP.

Abstract

While natural language processing tools have been developed extensively for some of the world's languages, a significant portion of the world's over 7000 languages are still neglected. One reason for this is that evaluation datasets do not yet cover a wide range of languages, including low-resource and endangered ones. We aim to address this issue by creating a text classification dataset encompassing a large number of languages, many of which currently have little to no annotated data available. We leverage parallel translations of the Bible to construct such a dataset by first developing applicable topics and employing a crowdsourcing tool to collect annotated data. By annotating the English side of the data and projecting the labels onto other languages through aligned verses, we generate text classification datasets for more than 1500 languages. We extensively benchmark several existing multilingual language models using our dataset. To facilitate the advancement of research in this area, we will release our dataset and code.

Figures (5)

• Figure 1: Tradeoff between K-$\alpha$ and the number of verses. Each dot in the plot stands for a threshold of the required minimum votes $\in \{3,4,5,6,7,8,9\}$ for a verse to be accepted.
• Figure 2: Confusion matrices of five-fold cross validation of XLM-R-Base and Glot500.
• Figure 3: Zero shot transfer learning: head languages (top), Glot500-only languages (middle) and tail languages (bottom). X-axis is the number of languages, y-axis presents four models. We split F1 scores into four ranges: 0-0.2, 0.2-0.4, 0.4-0.6 and 0.6-0.8.
• Figure 4: F1-differences of in-language learning for 1504 languages. We split the F1-differences between two models into six intervals: -0.4/-0.2, -0.2/-0.1, -0.1/0.0, 0.0/0.1,0.1/0.2, 0.2/0.4. Each bar represents the comparison between a pair of models.
• Figure 5: mTurk interface with English instructions and verse examples