Close to Human-Level Agreement: Tracing Journeys of Violent Speech in Incel Posts with GPT-4-Enhanced Annotations

TL;DR

The paper addresses the prevalence and temporal evolution of violent language in the Incel forum incels.is, leveraging a hybrid annotation pipeline that combines manual labeling with GPT-4 to scale classification. By annotating $N=33{,}028$ posts (about $15.7\%$ violent) and evaluating GPT-4 under various prompts and batch sizes, the study demonstrates substantial agreement with human coders (κ≈$0.54$–$0.62$) and achieves macro F1 around $0.63$–$0.67$, enabling large-scale moderation-oriented analysis. Time-based analyses reveal a forum-wide rise in violence over ~five years, a shift from directed to non-directed violence, and a decline in self-harm content after lengthy engagement, with nuanced dynamics at the individual level including a cool-off effect after inactivity. The work highlights GPT-4 as a cost-effective, scalable annotator for violent-language detection in extremist online communities, while noting the need for Incel-adapted taxonomy and further methodological comparisons to specialized hate-speech models. Overall, the approach provides practical tooling for moderation and foundational insight into the mechanisms of violent expression and potential radicalization in online forums.

Abstract

This study investigates the prevalence of violent language on incels.is. It evaluates GPT models (GPT-3.5 and GPT-4) for content analysis in social sciences, focusing on the impact of varying prompts and batch sizes on coding quality for the detection of violent speech. We scraped over 6.9M posts from incels.is and categorized a random sample into non-violent, explicitly violent, and implicitly violent content. Two human coders annotated 3,028 posts, which we used to tune and evaluate GPT-3.5 and GPT-4 models across different prompts and batch sizes regarding coding reliability. The best-performing GPT-4 model annotated an additional 30,000 posts for further analysis. Our findings indicate an overall increase in violent speech overtime on incels.is, both at the community and individual level, particularly among more engaged users. While directed violent language decreases, non-directed violent language increases, and self-harm content shows a decline, especially after 2.5 years of user activity. We find substantial agreement between both human coders (K = .65), while the best GPT-4 model yields good agreement with both human coders (K = 0.54 for Human A and K = 0.62 for Human B). Weighted and macro F1 scores further support this alignment. Overall, this research provides practical means for accurately identifying violent language at a large scale that can aid content moderation and facilitate next-step research into the causal mechanism and potential mitigations of violent expression and radicalization in communities like incels.is.

Figures (2)

• Figure 1: Linear Regression between time and share of violent posts. Numbers indicate the slope of the regression line, and stars indicate statistical significance.
• Figure 2: Linear Regression between time and category of directedness. Numbers indicate the slope of the regression line, and stars indicate statistical significance.