Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QQ: A Toolkit for Language Identifiers and Metadata

Wessel Poelman, Yiyi Chen, Miryam de Lhoneux

TL;DR

This work introduces QwanQwa, a light-weight Python toolkit for unified language metadata management that integrates multiple language resources into a single interface, provides convenient normalization and mapping between language identifiers, and affords a graph-based structure that enables traversal across families, regions, writing systems, and other linguistic attributes.

Abstract

The growing number of languages considered in multilingual NLP, including new datasets and tasks, poses challenges regarding properly and accurately reporting which languages are used and how. For example, datasets often use different language identifiers; some use BCP-47 (e.g. en_Latn), others use ISO 639-1 (en), and more linguistically oriented datasets use Glottocodes (stan1293). Mapping between identifiers is manageable for a few dozen languages, but becomes unscalable when dealing with thousands. We introduce QwanQwa, a light-weight Python toolkit for unified language metadata management. QQ integrates multiple language resources into a single interface, provides convenient normalization and mapping between language identifiers, and affords a graph-based structure that enables traversal across families, regions, writing systems, and other linguistic attributes. QQ serves both as (1) a simple "glue" library in multilingual NLP research to make working with many languages easier, and (2) as an intuitive way for exploring languages, such as finding related ones through shared scripts, regions or other metadata.

QQ: A Toolkit for Language Identifiers and Metadata

TL;DR

This work introduces QwanQwa, a light-weight Python toolkit for unified language metadata management that integrates multiple language resources into a single interface, provides convenient normalization and mapping between language identifiers, and affords a graph-based structure that enables traversal across families, regions, writing systems, and other linguistic attributes.

Abstract

The growing number of languages considered in multilingual NLP, including new datasets and tasks, poses challenges regarding properly and accurately reporting which languages are used and how. For example, datasets often use different language identifiers; some use BCP-47 (e.g. en_Latn), others use ISO 639-1 (en), and more linguistically oriented datasets use Glottocodes (stan1293). Mapping between identifiers is manageable for a few dozen languages, but becomes unscalable when dealing with thousands. We introduce QwanQwa, a light-weight Python toolkit for unified language metadata management. QQ integrates multiple language resources into a single interface, provides convenient normalization and mapping between language identifiers, and affords a graph-based structure that enables traversal across families, regions, writing systems, and other linguistic attributes. QQ serves both as (1) a simple "glue" library in multilingual NLP research to make working with many languages easier, and (2) as an intuitive way for exploring languages, such as finding related ones through shared scripts, regions or other metadata.
Paper Structure (18 sections, 3 equations, 4 figures, 5 tables)

This paper contains 18 sections, 3 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. QwanQwa
  3. Compiling the Database
  4. Using QQ
  5. Related Tools
  6. Case Studies
  7. Audit of Huggingface Datasets.
  8. Workflow in NLP Research.
  9. Linking Datasets.
  10. Graph-based Smoothness Analysis.
  11. Own-Language vs. Other-Language Colexified Concepts.
  12. Conclusion
  13. Data.
  14. Huggingface Audit.
  15. Details Colexification Analysis
  16. ...and 3 more sections

Figures (4)

  • Figure 1: QwanQwa (QQ) is a Python library that combines different language metadata sources into a unified and traversable graph structure.
  • Figure 2: Identifier usage across 1476 datasets with at least 10 languages in Huggingface datasets.
  • Figure 3: Own-Language vs. Other-Language Colexifications in Valence Similarity. (Cohen's d with Statistical Significance: *** $p < .001$, **$p < .01$, *$p < .05$).
  • Figure 4: UpSet plot showing identifier coverage per standard. The intersection between codes is based on the identifier resolving from QQ. ISO 639-2T is not included since it is a subset of ISO 639-3.