TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection

TL;DR

TRACE tackles hateful meme detection by integrating visually grounded context augmentation with a caption-aware, parameter-efficient fine-tuning strategy for CLIP's text encoder. A novel caption scorer, trained with a hate-relevance loss, selects the most informative image-caption pair, which is fused with image features via bidirectional cross-attention for final classification. On Hateful Memes, TRACE achieves state-of-the-art-level accuracy of approximately 0.807 and an F1 of about 0.806 while using a fraction of the compute of larger models, and it generalizes to the MultiOFF dataset (F1 ≈ 0.673). The framework also provides interpretability through its modular components and demonstrates robustness against benign confounders, with public release of code to facilitate further research and deployment considerations.

Abstract

Social media memes are a challenging domain for hate detection because they intertwine visual and textual cues into culturally nuanced messages. To tackle these challenges, we introduce TRACE, a hierarchical multimodal framework that leverages visually grounded context augmentation, along with a novel caption-scoring network to emphasize hate-relevant content, and parameter-efficient fine-tuning of CLIP's text encoder. Our experiments demonstrate that selectively fine-tuning deeper text encoder layers significantly enhances performance compared to simpler projection-layer fine-tuning methods. Specifically, our framework achieves state-of-the-art accuracy (0.807) and F1-score (0.806) on the widely-used Hateful Memes dataset, matching the performance of considerably larger models while maintaining efficiency. Moreover, it achieves superior generalization on the MultiOFF offensive meme dataset (F1-score 0.673), highlighting robustness across meme categories. Additional analyses confirm that robust visual grounding and nuanced text representations significantly reduce errors caused by benign confounders. We publicly release our code to facilitate future research.

Figures (6)

• Figure 1: Illustration of benign confounders (absent from the dataset) as noted by kiela2020hateful. (left) meme, (centre) image confounder, (right) text confounder.
• Figure 2: Proposed hierarchical architecture of the TRACE framework for Hateful Meme detection. The process includes: (1) Visual Grounding using RAM and GroundingDINO; (2) Context Augmentation where LVLMs generate captions incorporating grounded visual details and original text; (3) A VLM layer (such as CLIP) encoding the image and augmented text; (4) A novel Caption Scorer selecting the most relevant caption; and (5) Joint fine-tuning the text encoder and caption scorer for final classification. Frozen components () remain untrained.
• Figure 3: Parameter-Efficient Fine-tuning of the CLIP text encoder within TRACE. This diagram illustrates the strategy where only the last $n$-layers (indicated by symbol) of the text encoder (comprising Layer Norm, Multi-Head Attention, and Feed Forward blocks) are fine-tuned, alongside the Caption Scorer. This minimizes trainable parameters to prevent overfitting while adapting the model for hate detection.
• Figure 4: F1 Score vs. the number of fine-tuned text encoder layers ($n$). Illustrates the impact of increasing the number of trainable layers ($n$, from $1$ to $5$) in the text encoder on F1 performance for CLIP-ViT-L/14, CLIP-XLM-R-ViT-H/14, and SigLIP2-L/16-384 models within the TRACE framework on the FHM dataset. Fine-tuning $4$ layers achieves the best performance for all three models.
• Figure 5: TRACE predictions using CLIP-XLM-R-ViT-H-14. (a) A correct classification where subtle hateful objectification is identified. (b) A correctly identified non-hateful meme.