Modularized Networks for Few-shot Hateful Meme Detection

TL;DR

The paper tackles hateful meme detection in low-resource settings by proposing Mod-HATE, a modularized network that learns a library of LoRA modules from related tasks and composes them with a small trainable module composer. The approach uses an image-to-text converter to handle multimodal inputs and integrates the composed module with a frozen LLM, achieving efficient inference and superior performance versus in-context baselines across three datasets in $4$- and $8$-shot regimes. Key contributions include the three LoRA modules aligned to hate speech understanding, meme comprehension, and hateful meme interpretation, the CMA-ES-based module composer, and comprehensive empirical validation with ablations and qualitative analyses. This framework demonstrates strong few-shot generalization for hateful meme detection and offers a scalable path for applying modular, parameter-efficient adaptation to other multimodal, low-resource tasks, with potential extensions to instance-dependent composition.

Abstract

In this paper, we address the challenge of detecting hateful memes in the low-resource setting where only a few labeled examples are available. Our approach leverages the compositionality of Low-rank adaptation (LoRA), a widely used parameter-efficient tuning technique. We commence by fine-tuning large language models (LLMs) with LoRA on selected tasks pertinent to hateful meme detection, thereby generating a suite of LoRA modules. These modules are capable of essential reasoning skills for hateful meme detection. We then use the few available annotated samples to train a module composer, which assigns weights to the LoRA modules based on their relevance. The model's learnable parameters are directly proportional to the number of LoRA modules. This modularized network, underpinned by LLMs and augmented with LoRA modules, exhibits enhanced generalization in the context of hateful meme detection. Our evaluation spans three datasets designed for hateful meme detection in a few-shot learning context. The proposed method demonstrates superior performance to traditional in-context learning, which is also more computationally intensive during inference.We then use the few available annotated samples to train a module composer, which assigns weights to the LoRA modules based on their relevance. The model's learnable parameters are directly proportional to the number of LoRA modules. This modularized network, underpinned by LLMs and augmented with LoRA modules, exhibits enhanced generalization in the context of hateful meme detection. Our evaluation spans three datasets designed for hateful meme detection in a few-shot learning context. The proposed method demonstrates superior performance to traditional in-context learning, which is also more computationally intensive during inference.

Figures (3)

• Figure 1: Comparison of standard in-context learning and our proposed modularized networks for few-shot hateful meme detection.
• Figure 2: Overview of the proposed Mod-HATE model. It consists of three steps: 1) LoRA module learning from relevant tasks to obtain essential skills for hateful meme detection; 2) training a module composer with few-shot training data to learn importance scores assigned over each LoRA module; 3) the construction of modularized networks by integrating the composition of learned LoRA modules with frozen LLMs.
• Figure 3: Examples of (a) meme comprehension and (b) hateful meme interpretation tasks.