From Shallow Humor to Metaphor: Towards Label-Free Harmful Meme Detection via LMM Agent Self-Improvement

TL;DR

This work tackles harmful meme detection without relying on labeled data, addressing two core challenges: the need for large annotations and rapid evolution of harmful content. It introduces ALARM, a label-free framework that uses confidence-based explicit meme identification to pseudo-label easy memes and forms contrastive pairs to drive a self-improving LMM agent that gathers experiences and distills high-level references. The agent then applies these references to detect more subtle, emerging memes, demonstrated across three diverse datasets where ALARM outperforms many label-driven baselines and shows strong generalization and cross-model transferability. Overall, ALARM provides a scalable, training-free approach that adapts quickly to novel meme forms in dynamic online environments, highlighting the potential of unlabeled data and autonomous self-improvement in harmful-content detection.

Abstract

The proliferation of harmful memes on online media poses significant risks to public health and stability. Existing detection methods heavily rely on large-scale labeled data for training, which necessitates substantial manual annotation efforts and limits their adaptability to the continually evolving nature of harmful content. To address these challenges, we present ALARM, the first lAbeL-free hARmful Meme detection framework powered by Large Multimodal Model (LMM) agent self-improvement. The core innovation of ALARM lies in exploiting the expressive information from "shallow" memes to iteratively enhance its ability to tackle more complex and subtle ones. ALARM consists of a novel Confidence-based Explicit Meme Identification mechanism that isolates the explicit memes from the original dataset and assigns them pseudo-labels. Besides, a new Pairwise Learning Guided Agent Self-Improvement paradigm is introduced, where the explicit memes are reorganized into contrastive pairs (positive vs. negative) to refine a learner LMM agent. This agent autonomously derives high-level detection cues from these pairs, which in turn empower the agent itself to handle complex and challenging memes effectively. Experiments on three diverse datasets demonstrate the superior performance and strong adaptability of ALARM to newly evolved memes. Notably, our method even outperforms label-driven methods. These results highlight the potential of label-free frameworks as a scalable and promising solution for adapting to novel forms and topics of harmful memes in dynamic online environments.

Figures (9)

• Figure 1: Concept diagram of our framework ALARM, where the LMM agent achieves self-improvement by distilling the "powerful experiences" from the explicit memes, therefore delivering enhanced detections on the subtle ones.
• Figure 2: The overall framework of ALARM. (1) The original meme dataset are divided into both explicit and subtle subsets through a Confidence-based Explicit Meme Identification mechanism. (2) The pseudo-labeled explicit memes in two categories are reorganized into contrastive pairs. And a self-driven LMM agent is positioned to gather experiences and derive references from these pairs via a Pairwise Learning Guided Agent Self-Improvement paradigm. (3) At inference, the self-generated references are leveraged to help LMM agent itself in tackling the subtle and challenge meme.
• Figure 3: The relationship between prediction probs threshold and detection accuracy on the FHM and MAMI datasets. The color and size of circles reflect the number of samples, with smaller and darker ones indicating fewer samples.
• Figure 4: Sensitivity analysis of hyper-parameters $\tau$ and $L$.
• Figure 5: Case study on experience gathering and reference refinement: A contrastive pair is analyzed, including both pseudo-benign and pseudo-harmful memes, alongside the experiences and references derived from this pair.