HateModerate: Testing Hate Speech Detectors against Content Moderation Policies

TL;DR

The paper tackles the gap between hate speech detectors and platform content policies by constructing HateModerate, a per-policy evaluation dataset aligned to Facebook's 41 guidelines. It details a six-step annotation pipeline combining human labor and GPT augmentation to produce per-policy hateful and non-hateful test items. Evaluations reveal substantial policy-conformity failures across state-of-the-art detectors, with OpenAI's Moderation API performing best and non-hateful cases posing major challenges. Fine-tuning models with HateModerate improves conformity to policies while preserving performance on existing test data, highlighting HateModerate's practical value for trustworthy moderation.

Abstract

To protect users from massive hateful content, existing works studied automated hate speech detection. Despite the existing efforts, one question remains: do automated hate speech detectors conform to social media content policies? A platform's content policies are a checklist of content moderated by the social media platform. Because content moderation rules are often uniquely defined, existing hate speech datasets cannot directly answer this question. This work seeks to answer this question by creating HateModerate, a dataset for testing the behaviors of automated content moderators against content policies. First, we engage 28 annotators and GPT in a six-step annotation process, resulting in a list of hateful and non-hateful test suites matching each of Facebook's 41 hate speech policies. Second, we test the performance of state-of-the-art hate speech detectors against HateModerate, revealing substantial failures these models have in their conformity to the policies. Third, using HateModerate, we augment the training data of a top-downloaded hate detector on HuggingFace. We observe significant improvement in the models' conformity to content policies while having comparable scores on the original test data. Our dataset and code can be found in the attachment.

Figures (5)

• Figure 1: Examples of community standards guidelines for hate speech fb_standard
• Figure 2: The workflow of data collection for Guideline 10 (Tier 1, Certain objects).
• Figure 3: The statistics of examples in each policy in our dataset
• Figure 4: We detect the failure rates for both hateful and non-hateful examples across each of the 41 policies in Facebook's community standards guidelines fb_standard. Perspective's threshold is 0.5; Perspective*'s threshold is 0.7. For each policy, the bars facing right show the failure rates of hateful examples; the bars facing left show the failure rates of non-hateful examples.
• Figure 5: The comparison of failure rates in each sub-categories of non-hateful examples