The Unseen Targets of Hate -- A Systematic Review of Hateful Communication Datasets

TL;DR

This systematic review interrogates the provenance and quality of hateful online-communication datasets, focusing on which target identities are included (conceptualized, operationalized) and which appear in actual data (detected). Using PRISMA-guided paper and dataset annotation plus a dictionary-based detection on a dataset sub-sample, it reveals a persistent English- and U.S.-centric bias, substantial linguistic diversification over time, and notable gaps for age, body image, and organizational targets. The study finds mismatches between declared targets and what datasets actually contain, with up to 16% of detected targets not previously conceptualized or operationalized, underscoring risks for classifier reliability and fairness. The authors argue for standardized reporting, target-aware data collection, and participatory, context-sensitive practices to improve dataset quality and the real-world impact of hate-speech moderation tools.

Abstract

Machine learning (ML)-based content moderation tools are essential to keep online spaces free from hateful communication. Yet, ML tools can only be as capable as the quality of the data they are trained on allows them. While there is increasing evidence that they underperform in detecting hateful communications directed towards specific identities and may discriminate against them, we know surprisingly little about the provenance of such bias. To fill this gap, we present a systematic review of the datasets for the automated detection of hateful communication introduced over the past decade, and unpack the quality of the datasets in terms of the identities that they embody: those of the targets of hateful communication that the data curators focused on, as well as those unintentionally included in the datasets. We find, overall, a skewed representation of selected target identities and mismatches between the targets that research conceptualizes and ultimately includes in datasets. Yet, by contextualizing these findings in the language and location of origin of the datasets, we highlight a positive trend towards the broadening and diversification of this research space.

Figures (14)

• Figure 1: Details of literature search and screening process using the PRISMA flow diagram. Notes: DL stands for Digital Library; I&E stands for Inclusion and Exclusion.
• Figure 2: The three types of targets studied in this work and the potential mismatches between them. We introduce a two-tier categorization of targets. First, we distinguish between conceptualized targets (i.e., those who are included in the explicit definition of hateful communication as a construct chosen by the researcher) and operationalized targets (i.e., those who are operationalized in the sampling, annotation and/or analysis of the dataset). Moreover, while conceptualized and operationalized targets are explicitly accounted for and typically described in the paper, the corresponding dataset may include other targets that are not: we call the latter detected targets. The figure depicts a mismatch between these three types of targets: the researcher has chosen a very broad conceptualization of hateful online communication encompassing rage, gender, and religion, but a narrow operationalization, which only aims to capture hate towards gender identities in the dataset; yet, ultimately, the final dataset may include also targets that were part neither of the conceptualization nor the operationalization, such as identities based on political ideology.
• Figure 3: Conceptualized and operationalized targets by year along with the distribution of datasets. Targeted refers to datasets that have explicitly mentioned at least one target in their construct definition (i.e., in the conceptualization phase) and/or publications in which the authors define concrete measures to ensure and validate the presence of at least one target group in the data (i.e., in the operationalization phase). Untargeted refers to all other datasets that do not meet these two criteria. Note that the data for 2022 is only partially available (as described in our Methods and Data Section).
• Figure 4: Single vs. multiple platforms as data sources over time. While most of the datasets are collected from a single source, around 2018 researchers are increasingly collecting data from multiple sources (i.e., two or more).
• Figure 5: Geographic distribution of researchers' affiliation that contributed datasets. Researchers affiliated with institutions located in the U.S. published the most datasets, followed by researchers from India and the United Kingdom institutions.