They Said Memes Were Harmless-We Found the Ones That Hurt: Decoding Jokes, Symbols, and Cultural References

TL;DR

CROSS-ALIGN+ addresses the core challenges in meme abuse detection by grounding multimodal signals in structured cultural knowledge, sharpening decision boundaries with adaptive contrastive fine-tuning via LoRA, and providing interpretable, evidence-driven explanations. The three-stage framework yields up to 17% relative improvements in macro-F1 across five benchmarks and eight LVLM backbones, while maintaining low computational overhead suitable for web-scale moderation. Stage I mitigates cultural blindness, Stage II reduces boundary ambiguity, and Stage III enhances interpretability through cascaded explanations anchored to external knowledge. This approach improves robustness to cultural nuance and symbolic abuse, offering practical benefits for safer online spaces and accountable moderation.

Abstract

Meme-based social abuse detection is challenging because harmful intent often relies on implicit cultural symbolism and subtle cross-modal incongruence. Prior approaches, from fusion-based methods to in-context learning with Large Vision-Language Models (LVLMs), have made progress but remain limited by three factors: i) cultural blindness (missing symbolic context), ii) boundary ambiguity (satire vs. abuse confusion), and iii) lack of interpretability (opaque model reasoning). We introduce CROSS-ALIGN+, a three-stage framework that systematically addresses these limitations: (1) Stage I mitigates cultural blindness by enriching multimodal representations with structured knowledge from ConceptNet, Wikidata, and Hatebase; (2) Stage II reduces boundary ambiguity through parameter-efficient LoRA adapters that sharpen decision boundaries; and (3) Stage III enhances interpretability by generating cascaded explanations. Extensive experiments on five benchmarks and eight LVLMs demonstrate that CROSS-ALIGN+ consistently outperforms state-of-the-art methods, achieving up to 17% relative F1 improvement while providing interpretable justifications for each decision.

Figures (4)

• Figure 1: Toy Example of Implicit Abuse in Memes. Illustrative samples showing how memes encode harmful intent through subtle cross-modal incongruence and cultural symbolism. Although humorous in isolation, cultural context exposes exclusionary or abusive meanings—e.g., the alt-right appropriation of "Pepe the Frog", gendered subservience, or racially charged imagery—highlighting the need for culturally grounded multimodal reasoning in CROSS-ALIGN+.
• Figure 2: Overview of CROSS-ALIGN+. The pipeline consists of three stages: (I) external knowledge grounding using ConceptNet, Wikidata, and Hatebase to enrich multimodal representations; (II) adaptive contrastive fine-tuning with LoRA to improve boundary discrimination; and (III) hierarchical multistage reasoning to generate interpretable explanations. CROSS-ALIGN+ outputs both abuse predictions and faithful justifications for transparent meme moderation.
• Figure 3: Interpretability–Accuracy Correlation. Scatter plots of macro-F1 versus Explanation Plausibility Score (EPS) for four LVLMs (LLaVA-V1.6, Qwen2.5-VL, InternVL2, and Gemma-VL). A strong positive correlation ($r \approx 0.84$) indicates that more plausible explanations are associated with higher predictive accuracy.
• Figure 4: Robustness Analysis. Macro-F1 drop (%) under four perturbations—text masking, image cropping, synonym substitution, and symbol substitution—across eight LVLMs. CROSS-ALIGN+ shows strong robustness, with under 4% average performance degradation across all settings.