Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Modal Transfer from Memes to Videos: Addressing Data Scarcity in Hateful Video Detection

Han Wang, Rui Yang Tan, Roy Ka-Wei Lee

TL;DR

This work tackles the scarcity of annotated hateful video data by leveraging abundant hateful memes as substitutes or augmentations for video data. A novel human-assisted re-annotation pipeline aligns meme labels with video definitions, enabling consistent cross-modal learning. Using two vision-language models and LoRA-based fine-tuning, the approach shows that re-annotated meme data can match video-trained performance and further improve results when combined with video data. On the MHC and HateMM benchmarks, the proposed method achieves state-of-the-art gains, offering a scalable pathway to robust hateful video detectors under data constraints, with code and datasets publicly available.

Abstract

Detecting hate speech in online content is essential to ensuring safer digital spaces. While significant progress has been made in text and meme modalities, video-based hate speech detection remains under-explored, hindered by a lack of annotated datasets and the high cost of video annotation. This gap is particularly problematic given the growing reliance on large models, which demand substantial amounts of training data. To address this challenge, we leverage meme datasets as both a substitution and an augmentation strategy for training hateful video detection models. Our approach introduces a human-assisted reannotation pipeline to align meme dataset labels with video datasets, ensuring consistency with minimal labeling effort. Using two state-of-the-art vision-language models, we demonstrate that meme data can substitute for video data in resource-scarce scenarios and augment video datasets to achieve further performance gains. Our results consistently outperform state-of-the-art benchmarks, showcasing the potential of cross-modal transfer learning for advancing hateful video detection. Dataset and code are available at https://github.com/Social-AI-Studio/CrossModalTransferLearning.

Cross-Modal Transfer from Memes to Videos: Addressing Data Scarcity in Hateful Video Detection

TL;DR

This work tackles the scarcity of annotated hateful video data by leveraging abundant hateful memes as substitutes or augmentations for video data. A novel human-assisted re-annotation pipeline aligns meme labels with video definitions, enabling consistent cross-modal learning. Using two vision-language models and LoRA-based fine-tuning, the approach shows that re-annotated meme data can match video-trained performance and further improve results when combined with video data. On the MHC and HateMM benchmarks, the proposed method achieves state-of-the-art gains, offering a scalable pathway to robust hateful video detectors under data constraints, with code and datasets publicly available.

Abstract

Detecting hate speech in online content is essential to ensuring safer digital spaces. While significant progress has been made in text and meme modalities, video-based hate speech detection remains under-explored, hindered by a lack of annotated datasets and the high cost of video annotation. This gap is particularly problematic given the growing reliance on large models, which demand substantial amounts of training data. To address this challenge, we leverage meme datasets as both a substitution and an augmentation strategy for training hateful video detection models. Our approach introduces a human-assisted reannotation pipeline to align meme dataset labels with video datasets, ensuring consistency with minimal labeling effort. Using two state-of-the-art vision-language models, we demonstrate that meme data can substitute for video data in resource-scarce scenarios and augment video datasets to achieve further performance gains. Our results consistently outperform state-of-the-art benchmarks, showcasing the potential of cross-modal transfer learning for advancing hateful video detection. Dataset and code are available at https://github.com/Social-AI-Studio/CrossModalTransferLearning.
Paper Structure (19 sections, 2 equations, 4 figures, 5 tables)

This paper contains 19 sections, 2 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formulation
  5. Re-Annotation for Label Alignment
  6. Overview of the Re-Annotation Process
  7. Model Prediction with Few-Shot Prompting
  8. Analysis of Re-Annotation Results
  9. Cross-Modality Fine-Tuning
  10. Experiments
  11. Experimental Settings
  12. Datasets
  13. Models
  14. Fine-Tuning Strategies
  15. Hyperparameter Analysis
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Comparison between our approach (bottom) and other solutions that performs Parameter-Efficient Fine-Tuning (PEFT) on VLM for hateful video detection.
  • Figure 2: Re-annotation pipeline of meme datasets.
  • Figure 3: Meme count distribution for FHM dataset, categorized by original labels, re-annotated labels by MHC, and HateMM definitions.
  • Figure 4: Meme count distribution for MAMI dataset, categorized by original labels, re-annotated labels by MHC, and HateMM definitions.