Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera-Gómez, Sara Hincapie-Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, Shamsuddeen Hassan Muhammad, Atnafu Lambebo Tonja, Hend Al-Khalifa, Nadia Ghezaiel Hammouda, Verrah Otiende, Tack Hwa Wong, Jakhongir Saydaliev, Melika Nobakhtian, Muhammad Ravi Shulthan Habibi, Chalamalasetti Kranti, Carol Muchemi, Khang Nguyen, Faisal Muhammad Adam, Luis Frentzen Salim, Reem Alqifari, Cynthia Amol, Joseph Marvin Imperial, Ilker Kesen, Ahmad Mustafid, Pavel Stepachev, Leshem Choshen, David Anugraha, Hamada Nayel, Seid Muhie Yimam, Vallerie Alexandra Putra, My Chiffon Nguyen, Azmine Toushik Wasi, Gouthami Vadithya, Rob van der Goot, Lanwenn ar C'horr, Karan Dua, Andrew Yates, Mithil Bangera, Yeshil Bangera, Hitesh Laxmichand Patel, Shu Okabe, Fenal Ashokbhai Ilasariya, Dmitry Gaynullin, Genta Indra Winata, Yiyuan Li, Juan Pablo Martínez, Amit Agarwal, Ikhlasul Akmal Hanif, Raia Abu Ahmad, Esther Adenuga, Filbert Aurelian Tjiaranata, Weerayut Buaphet, Michael Anugraha, Sowmya Vajjala, Benjamin Rice, Azril Hafizi Amirudin, Jesujoba O. Alabi, Srikant Panda, Yassine Toughrai, Bruhan Kyomuhendo, Daniel Ruffinelli, Akshata A, Manuel Goulão, Ej Zhou, Ingrid Gabriela Franco Ramirez, Cristina Aggazzotti, Konstantin Dobler, Jun Kevin, Quentin Pagès, Nicholas Andrews, Nuhu Ibrahim, Mattes Ruckdeschel, Amr Keleg, Mike Zhang, Casper Muziri, Saron Samuel, Sotaro Takeshita, Kun Kerdthaisong, Luca Foppiano, Rasul Dent, Tommaso Green, Ahmad Mustapha Wali, Kamohelo Makaaka, Vicky Feliren, Inshirah Idris, Hande Celikkanat, Abdulhamid Abubakar, Jean Maillard, Benoît Sagot, Thibault Clérice, Kenton Murray, Sarah Luger

TL;DR

CommonLID addresses the gap in reliable language identification for web data by introducing a large, human-annotated benchmark across 109 languages. The authors combine community-driven annotation with a careful sampling and quality-control process, and they rigorously evaluate eight LID models and several large language models across multiple datasets, highlighting that existing evaluations overstate web-domain performance. The work provides an open, extensible resource and a framework for fair, domain-aware evaluation, showing that no model simultaneously achieves high coverage and high accuracy across all domains. This benchmark is poised to drive the development of more robust LID systems and improve language representation in multilingual NLP, especially for long-tail languages, by exposing domain-specific weaknesses and enabling targeted improvements.

Abstract

Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heterogeneous web data often used to train multilingual language models. In this paper, we introduce CommonLID, a community-driven, human-annotated LID benchmark for the web domain, covering 109 languages. Many of the included languages have been previously under-served, making CommonLID a key resource for developing more representative high-quality text corpora. We show CommonLID's value by using it, alongside five other common evaluation sets, to test eight popular LID models. We analyse our results to situate our contribution and to provide an overview of the state of the art. In particular, we highlight that existing evaluations overestimate LID accuracy for many languages in the web domain. We make CommonLID and the code used to create it available under an open, permissive license.

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

TL;DR

CommonLID addresses the gap in reliable language identification for web data by introducing a large, human-annotated benchmark across 109 languages. The authors combine community-driven annotation with a careful sampling and quality-control process, and they rigorously evaluate eight LID models and several large language models across multiple datasets, highlighting that existing evaluations overstate web-domain performance. The work provides an open, extensible resource and a framework for fair, domain-aware evaluation, showing that no model simultaneously achieves high coverage and high accuracy across all domains. This benchmark is poised to drive the development of more robust LID systems and improve language representation in multilingual NLP, especially for long-tail languages, by exposing domain-specific weaknesses and enabling targeted improvements.

Abstract

Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heterogeneous web data often used to train multilingual language models. In this paper, we introduce CommonLID, a community-driven, human-annotated LID benchmark for the web domain, covering 109 languages. Many of the included languages have been previously under-served, making CommonLID a key resource for developing more representative high-quality text corpora. We show CommonLID's value by using it, alongside five other common evaluation sets, to test eight popular LID models. We analyse our results to situate our contribution and to provide an overview of the state of the art. In particular, we highlight that existing evaluations overestimate LID accuracy for many languages in the web domain. We make CommonLID and the code used to create it available under an open, permissive license.
Paper Structure (49 sections, 5 figures, 7 tables)

This paper contains 49 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Issues Affecting LID datasets
  3. Training Datasets
  4. Lack of Open Data
  5. Language Coverage
  6. Domain and Register
  7. Human Verification
  8. Evaluation Datasets
  9. Web Domain and Noisy Data
  10. Decreasing Availability
  11. Dataset Size
  12. Annotation Collection/Dataset Creation
  13. Sampling Common Crawl
  14. Recruitment
  15. Annotation Platform
  16. ...and 34 more sections

Figures (5)

  • Figure 1: Illustrative geographical distribution of the language varieties in CommonLID. The dot size represents the number of annotated lines, and colour represents language family.
  • Figure 2: Inference speed vs. F1 (76 core languages, combined datasets)
  • Figure 3: A screenshot of the annotation platform interface. The participant selects the language they want to annotate for.
  • Figure 4: A screenshot of the annotation platform interface. The participant has highlighted all text as Spanish.
  • Figure 5: A screenshot of the annotation platform interface. The participant has highlighted the English and Spanish text in the extract in different colours.