Concept Drift Guided LayerNorm Tuning for Efficient Multimodal Metaphor Identification

TL;DR

This paper tackles multimodal metaphor identification in memes by proposing CDGLT, a training-efficient framework that combines Concept Drift with SLERP on CLIP embeddings to generate drifted concept representations and a prompt-based LayerNorm Tuning pipeline for GPT-2. By fusing drifted and literal cross-modal features and restricting fine-tuning to LN layers plus a small prompt, the approach achieves state-of-the-art performance on MET-Meme while dramatically reducing training cost. The work introduces a novel prompt construction strategy and provides extensive ablations, demonstrating that Concept Drift improves figurative tasks like Metaphor Identification, while vanilla LN tuning remains strong for more direct tasks. Overall, CDGLT offers a scalable, efficient path toward accurate multimodal metaphor understanding with practical implications for meme analysis and cross-modal reasoning.

Abstract

Metaphorical imagination, the ability to connect seemingly unrelated concepts, is fundamental to human cognition and communication. While understanding linguistic metaphors has advanced significantly, grasping multimodal metaphors, such as those found in internet memes, presents unique challenges due to their unconventional expressions and implied meanings. Existing methods for multimodal metaphor identification often struggle to bridge the gap between literal and figurative interpretations. Additionally, generative approaches that utilize large language models or text-to-image models, while promising, suffer from high computational costs. This paper introduces \textbf{C}oncept \textbf{D}rift \textbf{G}uided \textbf{L}ayerNorm \textbf{T}uning (\textbf{CDGLT}), a novel and training-efficient framework for multimodal metaphor identification. CDGLT incorporates two key innovations: (1) Concept Drift, a mechanism that leverages Spherical Linear Interpolation (SLERP) of cross-modal embeddings from a CLIP encoder to generate a new, divergent concept embedding. This drifted concept helps to alleviate the gap between literal features and the figurative task. (2) A prompt construction strategy, that adapts the method of feature extraction and fusion using pre-trained language models for the multimodal metaphor identification task. CDGLT achieves state-of-the-art performance on the MET-Meme benchmark while significantly reducing training costs compared to existing generative methods. Ablation studies demonstrate the effectiveness of both Concept Drift and our adapted LN Tuning approach. Our method represents a significant step towards efficient and accurate multimodal metaphor understanding. The code is available: \href{https://github.com/Qianvenh/CDGLT}{https://github.com/Qianvenh/CDGLT}.

Figures (4)

• Figure 1: Concept Drift Phenomenon. Whether memes are metaphorical is closely related to the embedded text. (a) Before adding text: Snow White is about to take the apple. After adding text: The "cute boy" is likened to an apple that is about to be eaten, with the hope that new romantic interests aren't "toxic". (b) Before adding text: distracted boyfriend. After adding text: a joke about getting distracted from work responsibilities by looking at memes. It metaphorizes "Work that I should be doing" as neglected girlfriend and "Laughing at memes on teams" as the distracting beauty.
• Figure 2: The architecture of CDGLT which is implemented with feature extraction, Concept Drift modeling, and LN tuning of GPT-2 using our novel prompt design.
• Figure 3: Trend of accuracy and weighted F1-score with $\alpha$ of SLERP in Concept Drift on Metaphor Identification (MI) task.
• Figure 4: t-SNE visualizations of CLIP image, text and SLERP embeddings from MET-Meme training set. The left image shows the results for alpha = 0.2, while the right image shows the results for alpha = 0.8. The colors of the points represent different embedding types: image embeddings (blue), text embeddings (orange), and SLERP embeddings (green).