Beyond Binary Classification: Detecting Fine-Grained Sexism in Social Media Videos

TL;DR

The findings indicate that multimodal LLMs perform competitively with human annotators in identifying nuanced forms of sexism; however, they struggle to capture co-occurring sexist types when these are conveyed through visual cues.

Abstract

Online sexism appears in various forms, which makes its detection challenging. Although automated tools can enhance the identification of sexist content, they are often restricted to binary classification. Consequently, more subtle manifestations of sexism may remain undetected due to the lack of fine-grained, context-sensitive labels. To address this issue, we make the following contributions: (1) we present FineMuSe, a new multimodal sexism detection dataset in Spanish that includes both binary and fine-grained annotations; (2) we introduce a comprehensive hierarchical taxonomy that encompasses forms of sexism, non-sexism, and rhetorical devices of irony and humor; and (3) we evaluate a wide range of LLMs for both binary and fine-grained sexism detection. Our findings indicate that multimodal LLMs perform competitively with human annotators in identifying nuanced forms of sexism; however, they struggle to capture co-occurring sexist types when these are conveyed through visual cues.

Figures (8)

• Figure 1: Three-level hierarchical taxonomy for labeling sexism, non-sexism, and rhetorical devices in social media videos.
• Figure 2: Distribution of sexist and non-sexist categories across Part 1 and Part 2.
• Figure 3: Fleiss' Kappa values for sexist categories across Text and Video modalities in Part 1 and Part 2.
• Figure 4: Per-class F1 scores for fine-grained sexism detection across text-only and multimodal models. Results are split by dataset part (Part 1, Part 2). Horizontal dashed lines denote category averages across models. Multimodal models consistently outperform text-only models, especially in Part 2, though Objectification remains the hardest category across platforms.
• Figure 5: Correlation matrices for human annotations and Claude-3.7 Sonnet (V+L) predictions, based on the co-occurrence of sexist types.