xList-Hate: A Checklist-Based Framework for Interpretable and Generalizable Hate Speech Detection

TL;DR

Hate speech detection suffers from heterogeneous definitional criteria and annotation noise that hurt cross-dataset generalization. The authors propose xList-Hate, a diagnostic framework that decomposes the task into a fixed set of $10$ binary questions assessed by an LLM and aggregated by a lightweight decision tree, decoupling semantic judgment from final labeling. Empirical results show improved cross-dataset robustness and interpretability relative to zero-shot prompting and comparable or better robustness than supervised fine-tuning under domain shift. The approach offers transparent decision paths, resilience to annotation noise, and broad applicability beyond hate speech to other classification tasks requiring explicit conceptual reasoning. $\,$

Abstract

Hate speech detection is commonly framed as a direct binary classification problem despite being a composite concept defined through multiple interacting factors that vary across legal frameworks, platform policies, and annotation guidelines. As a result, supervised models often overfit dataset-specific definitions and exhibit limited robustness under domain shift and annotation noise. We introduce xList-Hate, a diagnostic framework that decomposes hate speech detection into a checklist of explicit, concept-level questions grounded in widely shared normative criteria. Each question is independently answered by a large language model (LLM), producing a binary diagnostic representation that captures hateful content features without directly predicting the final label. These diagnostic signals are then aggregated by a lightweight, fully interpretable decision tree, yielding transparent and auditable predictions. We evaluate it across multiple hate speech benchmarks and model families, comparing it against zero-shot LLM classification and in-domain supervised fine-tuning. While supervised methods typically maximize in-domain performance, we consistently improves cross-dataset robustness and relative performance under domain shift. In addition, qualitative analysis of disagreement cases provides evidence that the framework can be less sensitive to certain forms of annotation inconsistency and contextual ambiguity. Crucially, the approach enables fine-grained interpretability through explicit decision paths and factor-level analysis. Our results suggest that reframing hate speech detection as a diagnostic reasoning task, rather than a monolithic classification problem, provides a robust, explainable, and extensible alternative for content moderation.

Figures (3)

• Figure 1: Overview of the proposed checklist-based hate speech detection framework. Each input text $t$ is evaluated through a fixed set of ten diagnostic questions, answered independently by an LLM. The resulting feature vector is then passed to an interpretable decision layer, implemented as a lightweight decision tree, which performs the final classification.
• Figure 2: Checklist question importance across training datasets, model sizes (in billions of parameters), and model families. Each cell reports the mean normalized importance of a question in the induced decision trees, measured via impurity reduction.
• Figure 3: Decision tree learned on top of the checklist representations produced by mistral-24b, trained on the Measuring Hate Speech dataset. Each internal node corresponds to a checklist question, and each root-to-leaf path encodes a conjunction of diagnostic decisions that the model treats as sufficient for classification. Leaf nodes report the proportion of samples labeled as hate speech among the training instances that reach that leaf.