Effectiveness of Counter-Speech against Abusive Content: A Multidimensional Annotation and Classification Study

TL;DR

The paper tackles evaluating counter-speech effectiveness beyond surface metrics by proposing a six-dimension framework (Clarity, Evidence, Emotional Appeal, Rebuttal, Audience Adaptation, Fairness) and two classification strategies. It introduces a richly annotated resource by extending CONAN and Twitter HS/CS pairs with these six dimensions. It demonstrates that a dependency-based multi-task model achieves up to 0.96 F1 on combined data, outperforming baselines and revealing interdependencies among dimensions. Cross-domain evaluation exposes significant shifts, especially for Emotional Appeal, underscoring the need for domain-adaptive approaches.

Abstract

Counter-speech (CS) is a key strategy for mitigating online Hate Speech (HS), yet defining the criteria to assess its effectiveness remains an open challenge. We propose a novel computational framework for CS effectiveness classification, grounded in linguistics, communication and argumentation concepts. Our framework defines six core dimensions - Clarity, Evidence, Emotional Appeal, Rebuttal, Audience Adaptation, and Fairness - which we use to annotate 4,214 CS instances from two benchmark datasets, resulting in a novel linguistic resource released to the community. In addition, we propose two classification strategies, multi-task and dependency-based, achieving strong results (0.94 and 0.96 average F1 respectively on both expert- and user-written CS), outperforming standard baselines, and revealing strong interdependence among dimensions.

Figures (2)

• Figure 1: Examples of expert-written and user-written counter-speech.
• Figure 2: Learned Dependency Matrix obtained from Dependency_matrix_6e on the combined dataset with seed 42.