Can LLMs Evaluate What They Cannot Annotate? Revisiting LLM Reliability in Hate Speech Detection

TL;DR

This work investigates the reliability of LLMs as annotators for hate speech detection under a subjectivity-aware framework (xRR). It shows that LLMs diverge from human judgments at the instance level, though certain models (notably Nemo) partially reflect human annotation patterns and can preserve the relative ranking of classifiers. The study conducts two experiments: agreement analysis using traditional and xRR metrics, and ranking correlation to assess whether LLM-generated labels reproduce human model comparisons. The findings advise against treating LLMs as direct replacements for human annotators but support their use as scalable evaluators for comparing model performance in subjective NLP tasks, provided ranking stability is validated and calibration is considered.

Abstract

Hate speech spreads widely online, harming individuals and communities, making automatic detection essential for large-scale moderation, yet detecting it remains difficult. Part of the challenge lies in subjectivity: what one person flags as hate speech, another may see as benign. Traditional annotation agreement metrics, such as Cohen's $κ$, oversimplify this disagreement, treating it as an error rather than meaningful diversity. Meanwhile, Large Language Models (LLMs) promise scalable annotation, but prior studies demonstrate that they cannot fully replace human judgement, especially in subjective tasks. In this work, we reexamine LLM reliability using a subjectivity-aware framework, cross-Rater Reliability (xRR), revealing that even under fairer lens, LLMs still diverge from humans. Yet this limitation opens an opportunity: we find that LLM-generated annotations can reliably reflect performance trends across classification models, correlating with human evaluations. We test this by examining whether LLM-generated annotations preserve the relative ordering of model performance derived from human evaluation (i.e. whether models ranked as more reliable by human annotators preserve the same order when evaluated with LLM-generated labels). Our results show that, although LLMs differ from humans at the instance level, they reproduce similar ranking and classification patterns, suggesting their potential as proxy evaluators. While not a substitute for human annotators, they might serve as a scalable proxy for evaluation in subjective NLP tasks.

Figures (2)

• Figure 1: Pairwise Cohen's $\kappa$ agreement between human annotators and LLMs across the three datasets. Cells show $\kappa$ scores between pairs of raters.
• Figure 2: Per-target Missed Hate (%) for LLM majority.