Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classification

Siddhant Bikram Shah, Shuvam Shiwakoti, Maheep Chaudhary, Haohan Wang

TL;DR

This study introduces a novel dataset PrideMM comprising 5,063 text-embedded images associated with the LGBTQ+ Pride movement, thereby addressing a serious gap in existing resources and proposes a novel framework MemeCLIP for efficient downstream learning while preserving the knowledge of the pre-trained CLIP model.

Abstract

The complexity of text-embedded images presents a formidable challenge in machine learning given the need for multimodal understanding of multiple aspects of expression conveyed by them. While previous research in multimodal analysis has primarily focused on singular aspects such as hate speech and its subclasses, this study expands this focus to encompass multiple aspects of linguistics: hate, targets of hate, stance, and humor. We introduce a novel dataset PrideMM comprising 5,063 text-embedded images associated with the LGBTQ+ Pride movement, thereby addressing a serious gap in existing resources. We conduct extensive experimentation on PrideMM by using unimodal and multimodal baseline methods to establish benchmarks for each task. Additionally, we propose a novel framework MemeCLIP for efficient downstream learning while preserving the knowledge of the pre-trained CLIP model. The results of our experiments show that MemeCLIP achieves superior performance compared to previously proposed frameworks on two real-world datasets. We further compare the performance of MemeCLIP and zero-shot GPT-4 on the hate classification task. Finally, we discuss the shortcomings of our model by qualitatively analyzing misclassified samples. Our code and dataset are publicly available at: https://github.com/SiddhantBikram/MemeCLIP.

MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classification

TL;DR

This study introduces a novel dataset PrideMM comprising 5,063 text-embedded images associated with the LGBTQ+ Pride movement, thereby addressing a serious gap in existing resources and proposes a novel framework MemeCLIP for efficient downstream learning while preserving the knowledge of the pre-trained CLIP model.

Abstract

The complexity of text-embedded images presents a formidable challenge in machine learning given the need for multimodal understanding of multiple aspects of expression conveyed by them. While previous research in multimodal analysis has primarily focused on singular aspects such as hate speech and its subclasses, this study expands this focus to encompass multiple aspects of linguistics: hate, targets of hate, stance, and humor. We introduce a novel dataset PrideMM comprising 5,063 text-embedded images associated with the LGBTQ+ Pride movement, thereby addressing a serious gap in existing resources. We conduct extensive experimentation on PrideMM by using unimodal and multimodal baseline methods to establish benchmarks for each task. Additionally, we propose a novel framework MemeCLIP for efficient downstream learning while preserving the knowledge of the pre-trained CLIP model. The results of our experiments show that MemeCLIP achieves superior performance compared to previously proposed frameworks on two real-world datasets. We further compare the performance of MemeCLIP and zero-shot GPT-4 on the hate classification task. Finally, we discuss the shortcomings of our model by qualitatively analyzing misclassified samples. Our code and dataset are publicly available at: https://github.com/SiddhantBikram/MemeCLIP.
Paper Structure (22 sections, 6 equations, 3 figures, 9 tables)

This paper contains 22 sections, 6 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Datasets
  4. Multi-aspect Datasets
  5. Multimodal Frameworks
  6. Dataset
  7. Data Collection
  8. Data Annotation
  9. Annotation Guidelines
  10. Statistics and Inter-Annotator Agreement
  11. Methodology
  12. Experimental Results
  13. PrideMM Dataset
  14. HarMeme Dataset
  15. Ablation Study
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Samples of text-embedded images from the PrideMM dataset annotated across four aspect labels. The labels are in the form of {Hate, Target, Stance, Humor}. For Hate, {0, 1} correspond to No Hate and Hate respectively. For Target, {0, 1, 2, 3} correspond to hate targeted towards Undirected, Individual, Community, and Organization respectively. For Stance, {0, 1, 2} correspond to Neutral, Support, and Oppose respectively. For Humor, {0, 1} correspond to No Humor and Humor respectively.
  • Figure 2: An overview of our proposed framework, MemeCLIP. We use frozen CLIP image and text encoders to create representations for each image-text pair. These representations are passed through linear layers to disentangle the modalities in CLIP's shared embedding space. We implement Feature Adapters with residual connections for each modality to prevent overfitting. We use a cosine classifier to make MemeCLIP more robust to imbalanced data. We initialize classifier weights by using Semantic-Aware Initialization to further improve performance.
  • Figure 3: Examples of memes misclassified by MemeCLIP across four tasks. The labels are in the form of {Hate, Target, Stance, Humor}. Label details are outlined in Figure \ref{['fig:examples_1']}.