Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MemeLens: Multilingual Multitask VLMs for Memes

Ali Ezzat Shahroor, Mohamed Bayan Kmainasi, Abul Hasnat, Dimitar Dimitrov, Giovanni Da San Martino, Preslav Nakov, Firoj Alam

TL;DR

MemeLens tackles the fragmentation of meme understanding by unifying 38 public meme datasets across nine languages into a shared taxonomy of $20$ tasks within a multilingual, multimodal, multitask VLM. The paper demonstrates that multimodal training coupled with explanation augmentation yields broad, robust performance across tasks and languages, often matching or exceeding dataset-specific SOTA under controlled comparisons. A key finding is that unified multitask training generalizes better across datasets than single-dataset fine-tuning, though transfer remains uneven across languages and task types. The work provides comprehensive benchmarks, baselines, and resources to enable further research in cross-dataset meme understanding and transferable multimodal reasoning.

Abstract

Memes are a dominant medium for online communication and manipulation because meaning emerges from interactions between embedded text, imagery, and cultural context. Existing meme research is distributed across tasks (hate, misogyny, propaganda, sentiment, humour) and languages, which limits cross-domain generalization. To address this gap we propose MemeLens, a unified multilingual and multitask explanation-enhanced Vision Language Model (VLM) for meme understanding. We consolidate 38 public meme datasets, filter and map dataset-specific labels into a shared taxonomy of $20$ tasks spanning harm, targets, figurative/pragmatic intent, and affect. We present a comprehensive empirical analysis across modeling paradigms, task categories, and datasets. Our findings suggest that robust meme understanding requires multimodal training, exhibits substantial variation across semantic categories, and remains sensitive to over-specialization when models are fine-tuned on individual datasets rather than trained in a unified setting. We will make the experimental resources and datasets publicly available for the community.

MemeLens: Multilingual Multitask VLMs for Memes

TL;DR

MemeLens tackles the fragmentation of meme understanding by unifying 38 public meme datasets across nine languages into a shared taxonomy of tasks within a multilingual, multimodal, multitask VLM. The paper demonstrates that multimodal training coupled with explanation augmentation yields broad, robust performance across tasks and languages, often matching or exceeding dataset-specific SOTA under controlled comparisons. A key finding is that unified multitask training generalizes better across datasets than single-dataset fine-tuning, though transfer remains uneven across languages and task types. The work provides comprehensive benchmarks, baselines, and resources to enable further research in cross-dataset meme understanding and transferable multimodal reasoning.

Abstract

Memes are a dominant medium for online communication and manipulation because meaning emerges from interactions between embedded text, imagery, and cultural context. Existing meme research is distributed across tasks (hate, misogyny, propaganda, sentiment, humour) and languages, which limits cross-domain generalization. To address this gap we propose MemeLens, a unified multilingual and multitask explanation-enhanced Vision Language Model (VLM) for meme understanding. We consolidate 38 public meme datasets, filter and map dataset-specific labels into a shared taxonomy of tasks spanning harm, targets, figurative/pragmatic intent, and affect. We present a comprehensive empirical analysis across modeling paradigms, task categories, and datasets. Our findings suggest that robust meme understanding requires multimodal training, exhibits substantial variation across semantic categories, and remains sensitive to over-specialization when models are fine-tuned on individual datasets rather than trained in a unified setting. We will make the experimental resources and datasets publicly available for the community.
Paper Structure (51 sections, 2 figures, 13 tables)

This paper contains 51 sections, 2 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Meme benchmarks and shared tasks.
  4. Multilingual and multicultural meme understanding.
  5. Multitask meme modeling.
  6. Explanations, roles, and reasoning.
  7. VLMs and meme-specific adaptation.
  8. Comparison with prior work.
  9. Dataset
  10. Dataset Curation
  11. Dataset Statistics
  12. Explanation Augmentation
  13. Methodology
  14. Instruction Dataset
  15. Baselines
  16. ...and 36 more sections

Figures (2)

  • Figure 1: Overview of tasks and datasets in MemeLens. The unified task taxonomy and the mapping of each dataset to it are shown. Dataset-specific labels are mapped into a shared label space to support consistent multi-task training and cross-dataset evaluation.
  • Figure 2: Task--language coverage in MemeLens. Distribution of meme analysis tasks across languages.