Socio-Culturally Aware Evaluation Framework for LLM-Based Content Moderation
Shanu Kumar, Gauri Kholkar, Saish Mendke, Anubhav Sadana, Parag Agrawal, Sandipan Dandapat
TL;DR
The paper tackles the reliability gaps in LLM-based content moderation by introducing a socio-cultural aware evaluation framework that uses scalable persona-driven data generation. It pairs a diversity-focused generation pipeline with persona-driven augmentation to create a rich benchmark across hate speech, misinformation, sexual content, and self-harm, evaluated via zero-shot prompts on multiple LLMs. Key contributions include a scalable, low-annotation data generation method, a multi-task moderation benchmark across 300 targets, and a qualitative bias analysis that reveals how persona attributes shape generated content. The work advances fairer, more culturally aware moderation by exposing model weaknesses under socio-cultural variation and guiding future improvements in alignment and evaluation practices.
Abstract
With the growth of social media and large language models, content moderation has become crucial. Many existing datasets lack adequate representation of different groups, resulting in unreliable assessments. To tackle this, we propose a socio-culturally aware evaluation framework for LLM-driven content moderation and introduce a scalable method for creating diverse datasets using persona-based generation. Our analysis reveals that these datasets provide broader perspectives and pose greater challenges for LLMs than diversity-focused generation methods without personas. This challenge is especially pronounced in smaller LLMs, emphasizing the difficulties they encounter in moderating such diverse content.