Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledge

TL;DR

The paper tackles multimodal hateful meme detection by leveraging knowledge encoded in large multimodal models (LMMs). It prompts a frozen LMM to produce task-relevant semantic descriptions and elicited emotions, then combines four embeddings—image, embedded text, LMM semantic descriptions, and LMM emotions—via a frozen CLIP model to form rich meme representations. A novel LMM-based hard-mining objective is introduced, guiding a lightweight three-layer classifier with a cross-entropy loss and an auxiliary loss $\mathcal{L}_{HM}$ that uses LMM-predicted hard samples. The method achieves state-of-the-art results on Harm-C (LongCLIP) and PrideMM (CLIP), demonstrating the effectiveness of using LMM-generated knowledge in a fully multimodal, inference-only framework, while remaining more efficient than full LMM fine-tuning. Limitations include content subjectivity and computational costs, suggesting future work on smaller LMMs and multilingual support.

Abstract

Memes have become a dominant form of communication in social media in recent years. Memes are typically humorous and harmless, however there are also memes that promote hate speech, being in this way harmful to individuals and groups based on their identity. Therefore, detecting hateful content in memes has emerged as a task of critical importance. The need for understanding the complex interactions of images and their embedded text renders the hateful meme detection a challenging multimodal task. In this paper we propose to address the aforementioned task leveraging knowledge encoded in powerful Large Multimodal Models (LMM). Specifically, we propose to exploit LMMs in a two-fold manner. First, by extracting knowledge oriented to the hateful meme detection task in order to build strong meme representations. Specifically, generic semantic descriptions and emotions that the images along with their embedded texts elicit are extracted, which are then used to train a simple classification head for hateful meme detection. Second, by developing a novel hard mining approach introducing directly LMM-encoded knowledge to the training process, providing further improvements. We perform extensive experiments on two datasets that validate the effectiveness of the proposed method, achieving state-of-the-art performance. Our code and trained models are publicly available at: https://github.com/IDT-ITI/LMM-CLIP-meme.

Figures (1)

• Figure 1: The three steps of the proposed LMM-based training process for hateful meme detection. In the first step we prompt an LMM to extract semantic descriptions and elicited emotions for the memes, and we use a VLM to extract the corresponding embeddings in order to build the meme representations. The meme representations consist of four embeddings: 1) CLIP's image embeddings $\mathbf{e}^i$, 2) CLIP's embedded text embeddings $\mathbf{e}^t$, 3) LMM-generated semantic descriptions embeddings $\mathbf{e}^d$, and 4) LMM-generated elicited emotions embeddings $\mathbf{e}^m$. In the second step we prompt the LMM to identify hard samples. In the third step we train a simple classification head for hateful meme detection using a regular supervised loss and a new LMM-mined hard sample auxiliary loss, using the hard sample information obtained in the second step. At inference time, the first step is executed in order to produce the meme representation for a test meme, which is then is propagated to the trained classification head to predict if it is hateful or not.