Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection

Yingying Guo, Ke Zhang, Zirong Zeng

TL;DR

GatedCLIP is proposed, a Vision-Language model that enhances CLIP's multimodal capabilities with specialized architectural improvements for hateful memes detection and introduces learned projection heads that map CLIP embeddings to a task-optimized semantic space, a dynamic gated fusion mechanism that adaptively weights visual and textual features, and a contrastive learning objective that maintains cross-modal semantic alignment.

Abstract

Detecting hateful content in multimodal memes presents unique challenges, as harmful messages often emerge from the complex interplay between benign images and text. We propose GatedCLIP, a Vision-Language model that enhances CLIP's multimodal capabilities with specialized architectural improvements for hateful memes detection. Our approach introduces learned projection heads that map CLIP embeddings to a task-optimized semantic space, a dynamic gated fusion mechanism that adaptively weights visual and textual features, and a contrastive learning objective that maintains cross-modal semantic alignment. Experiments on the Hateful Memes dataset demonstrate that GatedCLIP achieves an AUROC of 0.66, substantially outperforming the CLIP baseline (AUROC 0.49) while maintaining computational efficiency with only 350K trainable parameters.

GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection

TL;DR

GatedCLIP is proposed, a Vision-Language model that enhances CLIP's multimodal capabilities with specialized architectural improvements for hateful memes detection and introduces learned projection heads that map CLIP embeddings to a task-optimized semantic space, a dynamic gated fusion mechanism that adaptively weights visual and textual features, and a contrastive learning objective that maintains cross-modal semantic alignment.

Abstract

Detecting hateful content in multimodal memes presents unique challenges, as harmful messages often emerge from the complex interplay between benign images and text. We propose GatedCLIP, a Vision-Language model that enhances CLIP's multimodal capabilities with specialized architectural improvements for hateful memes detection. Our approach introduces learned projection heads that map CLIP embeddings to a task-optimized semantic space, a dynamic gated fusion mechanism that adaptively weights visual and textual features, and a contrastive learning objective that maintains cross-modal semantic alignment. Experiments on the Hateful Memes dataset demonstrate that GatedCLIP achieves an AUROC of 0.66, substantially outperforming the CLIP baseline (AUROC 0.49) while maintaining computational efficiency with only 350K trainable parameters.
Paper Structure (22 sections, 9 equations, 3 figures, 1 table)

This paper contains 22 sections, 9 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Background: CLIP Architecture
  5. CLIP Baseline
  6. GatedCLIP Architecture
  7. Projection Heads
  8. Gated Fusion Mechanism
  9. Classification Head
  10. Training Objective
  11. Experimental Setup
  12. Dataset
  13. Implementation Details
  14. Evaluation Metrics
  15. Results
  16. ...and 7 more sections

Figures (3)

  • Figure 1: An example hateful meme from the dataset. The image shows a skunk with text "LOVE THE WAY YOU SMELL TODAY", where individually benign elements that together convey offensive content. This illustrates why multimodal understanding is crucial for hate detection.
  • Figure 2: GatedCLIP Architecture
  • Figure 3: Comparison of training dynamics between CLIP Baseline and GatedCLIP.