Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning

TL;DR

Formalized as $\mathcal{D}=\{(x_t, x_v, y)\}_{i=1}^N$ with $f:(x_t,x_v)\rightarrow y$, the paper tackles robust harmful meme detection under missing modalities. It introduces a shared representation learning baseline that independently projects each modality into a common latent space, enabling strong performance when text is unavailable and competitive results when both modalities are present. Using CLIP embeddings and a two-block FC network, the method achieves superior robustness to missing/text-noise conditions, especially on Harm-P at low text availability, and exhibits stability via ablation studies against fused representations. The work advances real-world applicability by maintaining detection accuracy in modal-incomplete scenarios, addressing a critical limitation of existing multimodal approaches.

Abstract

Internet memes are powerful tools for communication, capable of spreading political, psychological, and sociocultural ideas. However, they can be harmful and can be used to disseminate hate toward targeted individuals or groups. Although previous studies have focused on designing new detection methods, these often rely on modal-complete data, such as text and images. In real-world settings, however, modalities like text may be missing due to issues like poor OCR quality, making existing methods sensitive to missing information and leading to performance deterioration. To address this gap, in this paper, we present the first-of-its-kind work to comprehensively investigate the behavior of harmful meme detection methods in the presence of modal-incomplete data. Specifically, we propose a new baseline method that learns a shared representation for multiple modalities by projecting them independently. These shared representations can then be leveraged when data is modal-incomplete. Experimental results on two benchmark datasets demonstrate that our method outperforms existing approaches when text is missing. Moreover, these results suggest that our method allows for better integration of visual features, reducing dependence on text and improving robustness in scenarios where textual information is missing. Our work represents a significant step forward in enabling the real-world application of harmful meme detection, particularly in situations where a modality is absent.

Figures (1)

• Figure 1: Overview of our model. Each modality is independently selected through a switch and passed through a network of fully-connected layers to learn shared representations.