Large Vision-Language Models for Knowledge-Grounded Data Annotation of Memes

TL;DR

This work introduces CM50, a large-scale meme dataset focused on 50 templates to enable scalable meme understanding and analysis. It develops an automated annotation pipeline grounded in large vision-language models (notably GPT-4o) to generate image captions, meme captions, embedded text, and literary-device labels, aided by template knowledge from KnowYourMeme. It also presents a meme-text retrieval model (mtrCLIP) fine-tuned on CM50 to improve cross-modal retrieval, with evaluation on MemeCap and FigMemes showing meaningful gains in retrieval performance. While GPT-4o achieves near-human performance on several annotation tasks, challenges remain in accurately labeling complex figurative language; future work includes domain adaptation and pretraining a dedicated meme embedder to enhance scalability and robustness. Overall, the paper delivers a scalable framework and resources for knowledge-grounded meme annotation and retrieval, enabling large-scale analyses and downstream multimodal tasks in meme research.

Abstract

Memes have emerged as a powerful form of communication, integrating visual and textual elements to convey humor, satire, and cultural messages. Existing research has focused primarily on aspects such as emotion classification, meme generation, propagation, interpretation, figurative language, and sociolinguistics, but has often overlooked deeper meme comprehension and meme-text retrieval. To address these gaps, this study introduces ClassicMemes-50-templates (CM50), a large-scale dataset consisting of over 33,000 memes, centered around 50 popular meme templates. We also present an automated knowledge-grounded annotation pipeline leveraging large vision-language models to produce high-quality image captions, meme captions, and literary device labels overcoming the labor intensive demands of manual annotation. Additionally, we propose a meme-text retrieval CLIP model (mtrCLIP) that utilizes cross-modal embedding to enhance meme analysis, significantly improving retrieval performance. Our contributions include:(1) a novel dataset for large-scale meme study, (2) a scalable meme annotation framework, and (3) a fine-tuned CLIP for meme-text retrieval, all aimed at advancing the understanding and analysis of memes at scale.

Figures (5)

• Figure 1: Diagram of meme-text retrieval, illustrated with Recall@10 for clarity, where $n$ and $m$ represent the numbers of memes and texts respectively. Meme and text embeddings are normalized to compute cosine similarities. The light blue cells indicate true matches in the similarity matrix, while the pink cells highlight true matches within the ranked lists.
• Figure 2: Collage of memes found in CM50
• Figure 3: Example data annotation using our pipeline
• Figure 4: Left: Distribution of lengths (in tokens) for image captions, meme captions, and embedded text; Right: Statistical distribution of literary device labels across the dataset.
• Figure 5: CM50 dataset statistics of different templates present in our dataset.