WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge

TL;DR

A plug-in framework named WisdoM, to leverage the contextual world knowledge induced from the large vision-language models (LVLMs) for enhanced MSA, and to reduce the noise in the context, a training-free contextual fusion mechanism is designed.

Abstract

Sentiment analysis is rapidly advancing by utilizing various data modalities (e.g., text, image). However, most previous works relied on superficial information, neglecting the incorporation of contextual world knowledge (e.g., background information derived from but beyond the given image and text pairs) and thereby restricting their ability to achieve better multimodal sentiment analysis (MSA). In this paper, we proposed a plug-in framework named WisdoM, to leverage the contextual world knowledge induced from the large vision-language models (LVLMs) for enhanced MSA. WisdoM utilizes LVLMs to comprehensively analyze both images and corresponding texts, simultaneously generating pertinent context. To reduce the noise in the context, we also introduce a training-free contextual fusion mechanism. Experiments across diverse granularities of MSA tasks consistently demonstrate that our approach has substantial improvements (brings an average +1.96% F1 score among five advanced methods) over several state-of-the-art methods.

Figures (12)

• Figure 1: The simple schematic of our method. The sentiment polarity of Aleppo is negative, which is hard to directly predict by existing methods while our WisdoMpredicts correctly via incorporating context generated by the world knowledge-rich LVLMs.
• Figure 2: Template of task instruction.
• Figure 3: Detailed illustration of our proposed schema WisdoMwith a running example. Using ChatGPT to provide prompt templates. We then prompt LVLMs to generate context using the prompt templates with image and sentence. A training-free mechanism Contextual Fusion mitigates the noise in the context.
• Figure 4: Comparative winning rates of Our Context v.s. RAG-based methods on Twitter2015 and Twitter2017 benchmarks. We can see that our contexts are better than the knowledge provided by RAG and PKG.
• Figure 5: Effects of different types of world knowledge. We analyse the effect of different types of world knowledge by applying WisdoM to AoM. The orange dash line and blue dash line represent the F1-score of vanilla AoM on Twitter2015&2017 respectively.