Towards Better Inclusivity: A Diverse Tweet Corpus of English Varieties
Nhi Pham, Lachlan Pham, Adam L. Meyers
TL;DR
This work tackles bias from non-diverse English varieties in NLP by constructing a diverse tweet corpus across Asia, Africa, and Western countries. It introduces an annotation framework with six categories arranged along a pseudo-spectrum to surface distance from standard English and demonstrates strong inter-annotator agreement. The dataset comprises 170,800 tweets from 7 countries, annotated by regionally fluent researchers, and reveals sizable accuracy gaps in off-the-shelf language identifiers between western and non-western varieties. By providing a resource and methodology for expanding linguistic diversity in NLP data, the paper lays groundwork for more inclusive NLP and outlines scalable paths for future labeling and broader tasks.
Abstract
The prevalence of social media presents a growing opportunity to collect and analyse examples of English varieties. Whilst usage of these varieties was - and, in many cases, still is - used only in spoken contexts or hard-to-access private messages, social media sites like Twitter provide a platform for users to communicate informally in a scrapeable format. Notably, Indian English (Hinglish), Singaporean English (Singlish), and African-American English (AAE) can be commonly found online. These varieties pose a challenge to existing natural language processing (NLP) tools as they often differ orthographically and syntactically from standard English for which the majority of these tools are built. NLP models trained on standard English texts produced biased outcomes for users of underrepresented varieties. Some research has aimed to overcome the inherent biases caused by unrepresentative data through techniques like data augmentation or adjusting training models. We aim to address the issue of bias at its root - the data itself. We curate a dataset of tweets from countries with high proportions of underserved English variety speakers, and propose an annotation framework of six categorical classifications along a pseudo-spectrum that measures the degree of standard English and that thereby indirectly aims to surface the manifestations of English varieties in these tweets. Following best annotation practices, our growing corpus features 170,800 tweets taken from 7 countries, labeled by annotators who are from those countries and can communicate in regionally-dominant varieties of English. Our corpus highlights the accuracy discrepancies in pre-trained language identifiers between western English and non-western (i.e., less standard) English varieties. We hope to contribute to the growing literature identifying and reducing the implicit demographic discrepancies in NLP.