Using psychological theory to ground guidelines for the annotation of misogynistic language

TL;DR

This study tackles the problem of misogyny detection by grounding annotation guidelines in established psychological theory, addressing inconsistencies in prior datasets. It introduces a comprehensive, psychology-based coding scheme and an annotated Reddit dataset with substantial inter-rater reliability, then evaluates three large language models (LLMs) under three prompting conditions across three datasets, including an Ambivalent Sexism Inventory–derived set. The results show that the psychology-grounded guidelines improve misclassification performance relative to a prior scheme, though LLMs still exhibit built-in biases toward mainstream views and do not fully replicate human annotations. The work provides theory-driven resources and empirically demonstrates both the potential and current limitations of LLMs for misogyny detection, highlighting the need for continued refinement and bias mitigation. Overall, the paper advances a more principled approach to misogyny annotation and datasets while candidly addressing practical challenges in deploying LLMs for this task.

Abstract

Detecting misogynistic hate speech is a difficult algorithmic task. The task is made more difficult when decision criteria for what constitutes misogynistic speech are ungrounded in established literatures in psychology and philosophy, both of which have described in great detail the forms explicit and subtle misogynistic attitudes can take. In particular, the literature on algorithmic detection of misogynistic speech often rely on guidelines that are insufficiently robust or inappropriately justified -- they often fail to include various misogynistic phenomena or misrepresent their importance when they do. As a result, current misogyny detection coding schemes and datasets fail to capture the ways women experience misogyny online. This is of pressing importance: misogyny is on the rise both online and offline. Thus, the scientific community needs to have a systematic, theory informed coding scheme of misogyny detection and a corresponding dataset to train and test models of misogyny detection. To this end, we developed (1) a misogyny annotation guideline scheme informed by theoretical and empirical psychological research, (2) annotated a new dataset achieving substantial inter-rater agreement (kappa = 0.68) and (3) present a case study using Large Language Models (LLMs) to compare our coding scheme to a self-described "expert" misogyny annotation scheme in the literature. Our findings indicate that our guideline scheme surpasses the other coding scheme in the classification of misogynistic texts across 3 datasets. Additionally, we find that LLMs struggle to replicate our human annotator labels, attributable in large part to how LLMs reflect mainstream views of misogyny. We discuss implications for the use of LLMs for the purposes of misogyny detection.

Figures (4)

• Figure 1: Phases of the methodological workflow.
• Figure 2: Model estimates of the probability of classifying a text in the Guest et al. dataset correctly as a function of the interaction between the guideline scheme condition and the text class (misogynistic vs non-misogynistic). Bars represent the 95% Credible Intervals.
• Figure 3: Model estimates of the probability of classifying a text in our dataset correctly as a function of the interaction between the guideline scheme condition and the text class (misogynistic vs non-misogynistic). Bars represent the 95% Credible Intervals. We project the average prediction of the 3 LLMs for each text to show model fit to the data.
• Figure 4: Model estimates of the probability of classifying a text in the ASI dataset correctly as a function of the interaction between the guideline scheme condition and the text class (misogynistic vs non-misogynistic). Bars represent the 95% Credible Intervals.