Effectively Detecting and Responding to Online Harassment with Large Language Models

TL;DR

This work tackles harassment detection in private messaging using large language models and develops a two-LLM pipeline to label messages with context from prior conversations. It also introduces a simulated-response system that generates teen-language replies guided by intervention strategies. The LLM-based detector achieves high accuracy and outperforms a toxicity-focused baseline, while simulated responses are deemed more helpful than human replies, albeit slightly less natural. The study demonstrates the potential for privacy-preserving harassment interventions and outlines safety, bias, and scalability considerations for real-world deployment.

Abstract

Online harassment has been a persistent issue in the online space. Predominantly, research focused on online harassment in public social media platforms, while less is placed on private messaging platforms. To address online harassment on one private messaging platform, Instagram, we leverage the capabilities of Large Language Models (LLMs). To achieve this, we recruited human labelers to identify online harassment in an Instagram messages dataset. Using the previous conversation as context, we utilize an LLM pipeline to conduct large-scale labeling on Instagram messages and evaluate its performance against human labels. Then, we use LLM to generate and evaluate simulated responses to online harassment messages. We find that the LLM labeling pipeline is capable of identifying online harassment in private messages. By comparing human responses and simulated responses, we also demonstrate that our simulated responses are superior in helpfulness compared to original human responses.

Figures (2)

• Figure 1: LLM classification pipeline structure
• Figure 2: Simulated response pipeline structure