LLM-based Semantic Augmentation for Harmful Content Detection

TL;DR

This work tackles the challenge of detecting harmful content in social media by shifting from data augmentation to semantic augmentation using LLMs. It introduces LLM-based text cleaning to reduce caption noise and LLM-generated explanations or triggers to enrich input representations, evaluated on the SemEval 2024 Persuasive Memes and validated on Google Jigsaw Toxic Comments and Facebook Hateful Memes. The results show that zero-shot LLMs underperform on high-context, multi-label tasks, whereas combining text, machine captions, and explanations (and using triggers in sensitive domains) yields performance approaching or matching human-annotated data at a fraction of the cost. The findings underscore a scalable, cost-efficient pathway for improving social media classification, with clear guidance on when to employ explanations versus triggers and how to balance multimodal and textual cues in practice.

Abstract

Recent advances in large language models (LLMs) have demonstrated strong performance on simple text classification tasks, frequently under zero-shot settings. However, their efficacy declines when tackling complex social media challenges such as propaganda detection, hateful meme classification, and toxicity identification. Much of the existing work has focused on using LLMs to generate synthetic training data, overlooking the potential of LLM-based text preprocessing and semantic augmentation. In this paper, we introduce an approach that prompts LLMs to clean noisy text and provide context-rich explanations, thereby enhancing training sets without substantial increases in data volume. We systematically evaluate on the SemEval 2024 multi-label Persuasive Meme dataset and further validate on the Google Jigsaw toxic comments and Facebook hateful memes datasets to assess generalizability. Our results reveal that zero-shot LLM classification underperforms on these high-context tasks compared to supervised models. In contrast, integrating LLM-based semantic augmentation yields performance on par with approaches that rely on human-annotated data, at a fraction of the cost. These findings underscore the importance of strategically incorporating LLMs into machine learning (ML) pipeline for social media classification tasks, offering broad implications for combating harmful content online.

Figures (13)

• Figure 1: LLM-based Cleaning for BLIP-generated Caption.
• Figure 2: Overview of the experimental conditions.
• Figure 3: Performance visualization on the test set using different LLMs to generate meme explanations, with 5 runs per condition. GPT-4o maintains consistently strong performance across all conditions Text+Explanation (T+E), Text+Caption+Explanation (T+C+E), and Explanation Only (E), while Sonnet 3.5 performs well but shows slightly more variance. LLaMA 3.1 exhibits mixed results with high variance, particularly struggling in the Explanation-only (E) condition. The T+C+E condition generally yields the most stable performance across all models, suggesting the benefit of combining multiple information sources.
• Figure 4: Performance comparison across different model categories (3 runs). Our best configuration (LLM + decoder) achieves the highest performance at 62.2%, outperforming both task-specific baselines and zero-shot LLMs.
• Figure 5: Performance analysis across datasets and metrics. Our LLM + Encoder approach demonstrates strong performance across both tasks, achieving (a) high F1 and AUC scores on toxic comment detection comparable to specialized models, (b) superior AUC on hateful meme detection while maintaining competitive F1 scores, and (c) consistent performance across all metrics as shown in the heatmap visualization.