MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderation

TL;DR

MemeSense addresses the challenge of moderating memes that convey harm through subtle or contextual cues beyond explicit text. It introduces a three-stage, retrieval-augmented in-context learning framework that grounds meme interpretation in socially grounded commonsense cues and analogous reference memes, using cognitive shift vectors to adapt model representations. The approach yields superior semantic alignment and intervention quality across textless and textful memes, and on the ICMM benchmark, while providing insights into dataset construction, ablation, and model interpretability. The work advances real-world content moderation by enabling safer, context-aware intervention generation and offers open-source code and a new, nuanced dataset for further research.

Abstract

Online memes are a powerful yet challenging medium for content moderation, often masking harmful intent behind humor, irony, or cultural symbolism. Conventional moderation systems "especially those relying on explicit text" frequently fail to recognize such subtle or implicit harm. We introduce MemeSense, an adaptive framework designed to generate socially grounded interventions for harmful memes by combining visual and textual understanding with curated, semantically aligned examples enriched with commonsense cues. This enables the model to detect nuanced complexed threats like misogyny, stereotyping, or vulgarity "even in memes lacking overt language". Across multiple benchmark datasets, MemeSense outperforms state-of-the-art methods, achieving up to 35% higher semantic similarity and 9% improvement in BERTScore for non-textual memes, and notable gains for text-rich memes as well. These results highlight MemeSense as a promising step toward safer, more context-aware AI systems for real-world content moderation. Code and data available at: https://github.com/sayantan11995/MemeSense

Figures (4)

• Figure 1: Schematic diagram of MemeSense. Block 1 highlights the challenge of understanding memes in a zero-shot setting using MLLMs. Blocks 2 to 5 illustrate the key stages of our approach: (Block 2) Commonsense Parameter Generation, (Block 3) Exemplar Retrieval, (Block 4) Learning Cognitive Shift Vectors, and (Block 5) MemeSense Inference.
• Figure 2: Representative example of a harmful meme and the annotated commonsense parameters along with intervention.
• Figure 3: Memes can manifest harm in different ways, some rely solely on imagery to convey implicit messages, while others reinforce harm through accompanying text. This figure illustrates the three primary categories: (a) harmful memes without text, (b) harmful memes with text, and (c) non-harmful memes. Prior moderation efforts have disproportionately focused on text-based harmful memes, often overlooking the nuanced and context-dependent nature of purely visual memes.
• Figure 4: Representative examples of memes from each of the 15 commonsense harm categories.