FC-CONAN: An Exhaustively Paired Dataset for Robust Evaluation of Retrieval Systems

TL;DR

FC-CONAN (Fully Connected CONAN), the first dataset created by exhaustively considering all combinations of 45 English HS messages and 129 CNs, is introduced, enabling more faithful evaluation of counterspeech retrieval systems and facilitates detailed error analysis.

Abstract

Hate speech (HS) is a critical issue in online discourse, and one promising strategy to counter it is through the use of counter-narratives (CNs). Datasets linking HS with CNs are essential for advancing counterspeech research. However, even flagship resources like CONAN (Chung et al., 2019) annotate only a sparse subset of all possible HS-CN pairs, limiting evaluation. We introduce FC-CONAN (Fully Connected CONAN), the first dataset created by exhaustively considering all combinations of 45 English HS messages and 129 CNs. A two-stage annotation process involving nine annotators and four validators produces four partitions-Diamond, Gold, Silver, and Bronze-that balance reliability and scale. None of the labeled pairs overlap with CONAN, uncovering hundreds of previously unlabelled positives. FC-CONAN enables more faithful evaluation of counterspeech retrieval systems and facilitates detailed error analysis. The dataset is publicly available.

Figures (2)

• Figure 1: HIT RATIO@10 across the twelve systems (glossary in Section \ref{['sec:ExperimentalEvaluationEvaluationOfRecommendationSystems']}).
• Figure 2: MRR@10 of the twelve evaluated systems (see glossary in Section \ref{['sec:ExperimentalEvaluationEvaluationOfRecommendationSystems']}).

Theorems & Definitions (1)

• Example 1: An HS–CN pair from CONAN chung-etal-2019-conan