SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification
Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, Preslav Nakov
TL;DR
SOLID introduces a nine-million-tweet semi-supervised dataset for offensive language identification to overcome OLID's limited size and keyword bias. Using democratic co-training with an ensemble of PMI, FastText, LSTM, and BERT, the authors label SOLID from OLID seeds and show notable performance gains on OLID's Levels B and C, with a larger, well-annotated SOLID test set enabling stability analyses. The work also provides detailed dataset collection ethics, bias considerations, and a curse-word baseline analysis to delineate easy versus hard offensive tweets. SOLID is positioned as the official dataset of OffensEval 2020 and is intended to assist research in automatic moderation and reduction of toxic content while emphasizing careful use and potential risks.
Abstract
The widespread use of offensive content in social media has led to an abundance of research in detecting language such as hate speech, cyberbullying, and cyber-aggression. Recent work presented the OLID dataset, which follows a taxonomy for offensive language identification that provides meaningful information for understanding the type and the target of offensive messages. However, it is limited in size and it might be biased towards offensive language as it was collected using keywords. In this work, we present SOLID, an expanded dataset, where the tweets were collected in a more principled manner. SOLID contains over nine million English tweets labeled in a semi-supervised fashion. We demonstrate that using SOLID along with OLID yields sizable performance gains on the OLID test set for two different models, especially for the lower levels of the taxonomy.