RCLMuFN: Relational Context Learning and Multiplex Fusion Network for Multimodal Sarcasm Detection

TL;DR

RCLMuFN addresses multimodal sarcasm detection by modeling relational context between text and images without relying on graph structures. It fuses multimodal features through a Relational Context Learning module and a multiplex fusion mechanism, leveraging CLIP-based and traditional feature extractors. The approach achieves state-of-the-art performance on MMSD and MMSD 2.0, with ablative analysis confirming the contribution of each component. The work advances robust sarcasm understanding in dynamic contexts and offers a scalable, graph-free alternative for multimodal reasoning with practical implications for online safety and content moderation.

Abstract

Sarcasm typically conveys emotions of contempt or criticism by expressing a meaning that is contrary to the speaker's true intent. Accurate detection of sarcasm aids in identifying and filtering undesirable information on the Internet, thereby reducing malicious defamation and rumor-mongering. Nonetheless, the task of automatic sarcasm detection remains highly challenging for machines, as it critically depends on intricate factors such as relational context. Most existing multimodal sarcasm detection methods focus on introducing graph structures to establish entity relationships between text and images while neglecting to learn the relational context between text and images, which is crucial evidence for understanding the meaning of sarcasm. In addition, the meaning of sarcasm changes with the evolution of different contexts, but existing methods may not be accurate in modeling such dynamic changes, limiting the generalization ability of the models. To address the above issues, we propose a relational context learning and multiplex fusion network (RCLMuFN) for multimodal sarcasm detection. Firstly, we employ four feature extractors to comprehensively extract features from raw text and images, aiming to excavate potential features that may have been previously overlooked. Secondly, we utilize the relational context learning module to learn the contextual information of text and images and capture the dynamic properties through shallow and deep interactions. Finally, we employ a multiplex feature fusion module to enhance the generalization of the model by penetratingly integrating multimodal features derived from various interaction contexts. Extensive experiments on two multimodal sarcasm detection datasets show that our proposed method achieves state-of-the-art performance.

Figures (8)

• Figure 1: Our method can capture the relational context in the original text (Behaviour of getting up early for work is tightly linked to a love of work, suggesting that getting up at 3:30 am to go to work is not uncommon) and in the image (The black surroundings mean it's still dark).
• Figure 2: Relational Context Learning and Multiplex Fusion Network.
• Figure 3: CLIP-View Feature Fusion Module.
• Figure 4: Multiplex Feature Fusion Module.
• Figure 5: Visualization of sample cases using RCLM.