CrowdCounter: A benchmark type-specific multi-target counterspeech dataset

TL;DR

CrowdCounter introduces a large, crowd-sourced dataset of 3435 hate speech–counterspeech pairs spanning six counterspeech types to support type-specific generation. The authors design an annotation platform and sampling pipeline to maximize diversity and quality, sourcing data from HateXplain with a Gab focus, and pairing it with multi-model prompts (vanilla vs type-specific) across four language models. They evaluate generation and type-classification using a broad suite of referential, diversity, and quality metrics, finding that type-specific prompts improve relevance but can reduce language quality, while DialoGPT excels at following types. The work provides a practical benchmark for type-controlled counterspeech and highlights trade-offs between model size, instruction-following, and output quality, with implications for moderation tools and ethical deployment.

Abstract

Counterspeech presents a viable alternative to banning or suspending users for hate speech while upholding freedom of expression. However, writing effective counterspeech is challenging for moderators/users. Hence, developing suggestion tools for writing counterspeech is the need of the hour. One critical challenge in developing such a tool is the lack of quality and diversity of the responses in the existing datasets. Hence, we introduce a new dataset - CrowdCounter containing 3,425 hate speech-counterspeech pairs spanning six different counterspeech types (empathy, humor, questioning, warning, shaming, contradiction), which is the first of its kind. The design of our annotation platform itself encourages annotators to write type-specific, non-redundant and high-quality counterspeech. We evaluate two frameworks for generating counterspeech responses - vanilla and type-controlled prompts - across four large language models. In terms of metrics, we evaluate the responses using relevance, diversity and quality. We observe that Flan-T5 is the best model in the vanilla framework across different models. Type-specific prompts enhance the relevance of the responses, although they might reduce the language quality. DialoGPT proves to be the best at following the instructions and generating the type-specific counterspeech accurately.

Figures (1)

• Figure 1: The annotation platform which was used in MTurk.